2023-08-25T02:02:21.512 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-25T02:02:21.514 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922 --verbose --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} --name rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/orig.config.yaml 2023-08-25T02:02:21.581 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-25T02:02:21.667 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922 branch: wip-rishabh-2023aug3-b5 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} email: null first_in_suite: false job_id: '7378922' kernel: client: sha1: distro flavor: default kdb: true sha1: a6a69670761025e039dc5ae86619b0cf04c46dc2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-rishabh-2023aug3-b5 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mountpoint: /tmp/cephfs mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-2023aug3-b5 sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 owner: scheduled_rishabh@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 sleep_before_teardown: 0 suite: fs suite_branch: wip-rishabh-2023aug3-b5 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ba6b39e4f4f77fb95c747099b53db0467d435f56/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 500 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 449a1bc2027504e7b3c3d7b30fa4178906581da7 timestamp: 2023-08-25_01:50:32 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2224 2023-08-25T02:02:21.668 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ba6b39e4f4f77fb95c747099b53db0467d435f56/qa; will attempt to use it 2023-08-25T02:02:21.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ba6b39e4f4f77fb95c747099b53db0467d435f56/qa/tasks 2023-08-25T02:02:21.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-25T02:02:21.671 INFO:teuthology.task.internal:Checking packages... 2023-08-25T02:02:21.698 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'ba6b39e4f4f77fb95c747099b53db0467d435f56' 2023-08-25T02:02:21.698 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-25T02:02:21.698 INFO:teuthology.packaging:ref: None 2023-08-25T02:02:21.699 INFO:teuthology.packaging:tag: None 2023-08-25T02:02:21.699 INFO:teuthology.packaging:branch: wip-rishabh-2023aug3-b5 2023-08-25T02:02:21.699 INFO:teuthology.packaging:sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:02:21.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-2023aug3-b5 2023-08-25T02:02:22.107 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5703.gba6b39e4 2023-08-25T02:02:22.107 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-25T02:02:22.116 INFO:teuthology.task.internal:no buildpackages task found 2023-08-25T02:02:22.116 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-25T02:02:22.124 INFO:teuthology.task.internal:Saving configuration 2023-08-25T02:02:22.147 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-25T02:02:22.157 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T02:02:22.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.093492', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-25T02:02:22.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.094237', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-25T02:02:22.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.092559', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-25T02:02:22.232 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-25T02:02:22.238 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-08-25T02:02:22.238 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-08-25T02:02:22.238 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-08-25T02:02:22.238 INFO:teuthology.run_tasks:Running task console_log... 2023-08-25T02:02:22.340 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6e532de160>, signals=[15]) 2023-08-25T02:02:22.342 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-25T02:02:22.348 INFO:teuthology.task.internal:Opening connections... 2023-08-25T02:02:22.348 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:02:22.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:02:22.420 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-08-25T02:02:22.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:02:22.489 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2023-08-25T02:02:22.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:02:22.556 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-08-25T02:02:22.563 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2023-08-25T02:02:22.584 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2023-08-25T02:02:22.585 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-08-25T02:02:22.644 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Red Hat Enterprise Linux" 2023-08-25T02:02:22.644 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8.6 (Ootpa)" 2023-08-25T02:02:22.645 INFO:teuthology.orchestra.run.smithi089.stdout:ID="rhel" 2023-08-25T02:02:22.645 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="fedora" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8.6" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.redhat.com/" 2023-08-25T02:02:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-25T02:02:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-25T02:02:22.648 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-08-25T02:02:22.686 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-08-25T02:02:22.706 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-08-25T02:02:22.707 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-08-25T02:02:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-08-25T02:02:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)" 2023-08-25T02:02:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-08-25T02:02:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-08-25T02:02:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-25T02:02:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-25T02:02:22.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:02:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-25T02:02:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-25T02:02:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-25T02:02:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-25T02:02:22.770 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-08-25T02:02:22.793 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2023-08-25T02:02:22.814 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2023-08-25T02:02:22.815 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2023-08-25T02:02:22.874 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux" 2023-08-25T02:02:22.874 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-25T02:02:22.875 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/" 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:02:22.876 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-25T02:02:22.877 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-25T02:02:22.877 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-25T02:02:22.877 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-25T02:02:22.877 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-08-25T02:02:22.902 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-08-25T02:02:22.911 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-08-25T02:02:22.917 INFO:teuthology.task.internal:Checking for old test directory... 2023-08-25T02:02:22.917 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2023-08-25T02:02:22.920 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-08-25T02:02:22.922 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2023-08-25T02:02:22.941 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-08-25T02:02:22.946 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-08-25T02:02:22.947 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2023-08-25T02:02:22.979 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-08-25T02:02:22.982 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2023-08-25T02:02:23.016 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-08-25T02:02:23.090 INFO:teuthology.run_tasks:Running task kernel... 2023-08-25T02:02:23.105 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'} 2023-08-25T02:02:23.106 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-08-25T02:02:23.106 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-08-25T02:02:23.106 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-25T02:02:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:23.126 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-08-25T02:02:23.184 INFO:teuthology.orchestra.run.smithi089.stdout:4.18.0-372.9.1.el8.x86_64 2023-08-25T02:02:23.185 INFO:teuthology.task.kernel:Running kernel on smithi089: 4.18.0-372.9.1.el8.x86_64 2023-08-25T02:02:23.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2023-08-25T02:02:24.391 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:02:24.391 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-08-25T02:02:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 9.7 kB/s | 3.5 kB 00:00 2023-08-25T02:02:37.132 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 16 MB 00:11 2023-08-25T02:02:40.616 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 553 kB/s | 24 kB 00:00 2023-08-25T02:02:41.268 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-08-25T02:02:41.330 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-08-25T02:02:41.331 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-08-25T02:02:41.332 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-08-25T02:02:41.462 DEBUG:teuthology.orchestra.run.smithi089:> echo no | sudo yum reinstall kernel || true 2023-08-25T02:02:41.794 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:02:41.794 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-08-25T02:02:42.053 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:02 ago on Fri 25 Aug 2023 02:02:40 AM UTC. 2023-08-25T02:02:42.093 INFO:teuthology.orchestra.run.smithi089.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-25T02:02:42.130 INFO:teuthology.orchestra.run.smithi089.stderr:Error: No packages marked for reinstall. 2023-08-25T02:02:42.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum reinstall -y kernel || true 2023-08-25T02:02:42.509 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:02:42.509 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-08-25T02:02:42.760 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:02 ago on Fri 25 Aug 2023 02:02:40 AM UTC. 2023-08-25T02:02:42.799 INFO:teuthology.orchestra.run.smithi089.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-25T02:02:42.828 INFO:teuthology.orchestra.run.smithi089.stderr:Error: No packages marked for reinstall. 2023-08-25T02:02:42.876 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2023-08-25T02:02:42.956 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-08-25T02:02:42.957 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-08-25T02:02:42.957 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-08-25T02:02:42.957 DEBUG:teuthology.parallel:result is None 2023-08-25T02:02:42.957 INFO:teuthology.run_tasks:Running task internal.base... 2023-08-25T02:02:42.964 INFO:teuthology.task.internal:Creating test directory... 2023-08-25T02:02:42.964 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-25T02:02:42.967 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-25T02:02:42.969 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-25T02:02:42.991 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-08-25T02:02:42.999 INFO:teuthology.run_tasks:Running task internal.archive... 2023-08-25T02:02:43.009 INFO:teuthology.task.internal:Creating archive directory... 2023-08-25T02:02:43.010 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-25T02:02:43.022 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-25T02:02:43.028 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-25T02:02:43.068 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-08-25T02:02:43.075 INFO:teuthology.task.internal:Enabling coredump saving... 2023-08-25T02:02:43.075 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-25T02:02:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:43.114 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-25T02:02:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:43.133 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-25T02:02:43.135 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-25T02:02:43.156 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-25T02:02:43.164 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.184 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.195 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.213 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.242 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.260 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T02:02:43.261 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-08-25T02:02:43.268 INFO:teuthology.task.internal:Configuring sudo... 2023-08-25T02:02:43.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-25T02:02:43.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-25T02:02:43.273 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-25T02:02:43.302 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-08-25T02:02:43.311 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-08-25T02:02:43.311 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-25T02:02:43.338 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-25T02:02:43.341 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-25T02:02:43.359 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:43.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:43.484 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:43.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:43.608 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:02:43.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-25T02:02:43.676 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:43.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:43.773 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:43.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:43.901 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:02:43.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-25T02:02:43.969 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:44.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T02:02:44.079 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:44.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T02:02:44.173 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:02:44.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-25T02:02:44.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2023-08-25T02:02:44.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-08-25T02:02:44.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2023-08-25T02:02:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:02:44.319 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:02:44.387 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:02:44.795 INFO:teuthology.run_tasks:Running task internal.timer... 2023-08-25T02:02:44.803 INFO:teuthology.task.internal:Starting timer... 2023-08-25T02:02:44.804 INFO:teuthology.run_tasks:Running task pcp... 2023-08-25T02:02:44.814 INFO:teuthology.run_tasks:Running task selinux... 2023-08-25T02:02:44.865 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-08-25T02:02:44.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2023-08-25T02:02:44.924 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2023-08-25T02:02:44.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-08-25T02:02:45.001 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2023-08-25T02:02:45.003 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2023-08-25T02:02:45.059 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2023-08-25T02:02:45.060 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-08-25T02:02:45.061 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2023-08-25T02:02:45.082 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2023-08-25T02:02:45.083 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-08-25T02:02:45.105 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-08-25T02:02:45.105 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2023-08-25T02:02:45.130 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2023-08-25T02:02:45.130 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2023-08-25T02:02:45.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:02:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:45.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:02:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:45.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:02:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:02:45.213 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-08-25T02:02:45.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2023-08-25T02:02:45.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-08-25T02:02:45.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2023-08-25T02:02:45.311 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-08-25T02:02:45.321 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-08-25T02:02:45.321 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-08-25T02:02:45.342 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-08-25T02:02:45.344 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-08-25T02:13:45.626 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2023-08-25T02:13:45.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-08-25T02:13:45.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:13:45.728 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-08-25T02:13:45.803 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-08-25T02:13:45.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-25T02:13:45.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:13:45.877 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-08-25T02:13:45.943 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-25T02:13:45.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-08-25T02:13:45.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T02:13:46.015 DEBUG:teuthology.orchestra.run.smithi122:> true 2023-08-25T02:13:46.085 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-08-25T02:13:46.085 INFO:teuthology.run_tasks:Running task clock... 2023-08-25T02:13:46.096 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-08-25T02:13:46.097 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-25T02:13:46.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:13:46.101 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-25T02:13:46.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:13:46.104 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-25T02:13:46.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:13:46.135 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-25T02:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-25T02:13:46.152 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-25T02:13:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-25T02:13:46.162 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-25T02:13:46.178 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-25T02:13:46.214 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-08-25T02:13:46.222 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2023-08-25T02:13:46.226 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-08-25T02:13:46.233 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2023-08-25T02:13:46.242 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-25T02:13:46.245 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2023-08-25T02:13:46.248 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-25T02:13:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-25T02:13:46.257 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2023-08-25T02:13:46.263 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-25T02:13:46.272 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-25T02:13:46.286 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-25T02:13:46.329 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2023-08-25T02:13:46.333 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:13:46.333 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2023-08-25T02:13:46.333 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.334 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.334 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.334 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.336 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-08-25T02:13:46.339 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:13:46.340 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-08-25T02:13:46.340 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.340 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.340 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.340 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.358 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2023-08-25T02:13:46.361 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:13:46.361 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2023-08-25T02:13:46.362 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.362 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.362 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.362 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:13:46.363 INFO:teuthology.run_tasks:Running task pexec... 2023-08-25T02:13:48.636 INFO:teuthology.task.pexec:Executing custom commands... 2023-08-25T02:13:48.637 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:13:48.637 DEBUG:teuthology.orchestra.run.smithi089:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-08-25T02:13:48.638 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2023-08-25T02:13:48.638 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-08-25T02:13:48.638 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi122.front.sepia.ceph.com 2023-08-25T02:13:48.638 DEBUG:teuthology.orchestra.run.smithi122:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-08-25T02:13:49.019 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-08-25T02:13:49.020 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-08-25T02:13:49.022 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:13:50.618 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:21 ago on Fri 25 Aug 2023 02:11:29 AM UTC. 2023-08-25T02:13:50.705 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:12 ago on Fri 25 Aug 2023 02:11:38 AM UTC. 2023-08-25T02:13:50.739 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:03 ago on Fri 25 Aug 2023 02:11:47 AM UTC. 2023-08-25T02:13:52.331 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-08-25T02:13:52.332 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-08-25T02:13:52.332 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2023-08-25T02:13:52.332 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout:Resetting modules: 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-08-25T02:13:52.333 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:13:52.453 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout:Resetting modules: 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools 2023-08-25T02:13:52.454 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:13:52.455 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-08-25T02:13:52.455 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-08-25T02:13:52.455 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:13:52.583 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-08-25T02:13:52.584 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-08-25T02:13:52.584 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout:Resetting modules: 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-08-25T02:13:52.585 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:13:53.289 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:23 ago on Fri 25 Aug 2023 02:11:29 AM UTC. 2023-08-25T02:13:53.424 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:05 ago on Fri 25 Aug 2023 02:11:47 AM UTC. 2023-08-25T02:13:53.573 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:15 ago on Fri 25 Aug 2023 02:11:38 AM UTC. 2023-08-25T02:14:01.647 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-08-25T02:14:02.100 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:03.804 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:34 ago on Fri 25 Aug 2023 02:11:29 AM UTC. 2023-08-25T02:14:03.966 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-08-25T02:14:04.444 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:05.528 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-08-25T02:14:05.618 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================================ 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================================ 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout:Installing group/module packages: 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-08-25T02:14:05.619 INFO:teuthology.orchestra.run.smithi089.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-08-25T02:14:05.620 INFO:teuthology.orchestra.run.smithi089.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout:Installing module profiles: 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools/common 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling module streams: 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout: container-tools rhel8 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:05.621 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-08-25T02:14:05.622 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================================ 2023-08-25T02:14:05.622 INFO:teuthology.orchestra.run.smithi089.stdout:Install 10 Packages 2023-08-25T02:14:05.622 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:05.623 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 18 M 2023-08-25T02:14:05.623 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 65 M 2023-08-25T02:14:05.623 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-08-25T02:14:05.881 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-08-25T02:14:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 68 kB/s | 25 kB 00:00 2023-08-25T02:14:06.002 INFO:teuthology.orchestra.run.smithi089.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 107 kB/s | 41 kB 00:00 2023-08-25T02:14:06.029 INFO:teuthology.orchestra.run.smithi089.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 231 kB/s | 94 kB 00:00 2023-08-25T02:14:06.094 INFO:teuthology.orchestra.run.smithi089.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 453 kB/s | 49 kB 00:00 2023-08-25T02:14:06.177 INFO:teuthology.orchestra.run.smithi089.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.9 MB/s | 524 kB 00:00 2023-08-25T02:14:06.268 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:18 ago on Fri 25 Aug 2023 02:11:47 AM UTC. 2023-08-25T02:14:06.288 INFO:teuthology.orchestra.run.smithi089.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 779 kB/s | 149 kB 00:00 2023-08-25T02:14:06.344 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:06.388 INFO:teuthology.orchestra.run.smithi089.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-08-25T02:14:06.412 INFO:teuthology.orchestra.run.smithi089.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 910 kB/s | 213 kB 00:00 2023-08-25T02:14:06.637 INFO:teuthology.orchestra.run.smithi089.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 20 MB/s | 7.0 MB 00:00 2023-08-25T02:14:06.789 INFO:teuthology.orchestra.run.smithi089.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 5.5 MB/s | 2.2 MB 00:00 2023-08-25T02:14:06.789 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:14:06.790 INFO:teuthology.orchestra.run.smithi089.stdout:Total 16 MB/s | 18 MB 00:01 2023-08-25T02:14:06.992 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-08-25T02:14:07.010 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-08-25T02:14:07.016 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-08-25T02:14:07.193 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-08-25T02:14:07.193 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-08-25T02:14:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-08-25T02:14:07.909 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-08-25T02:14:07.957 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:29 ago on Fri 25 Aug 2023 02:11:38 AM UTC. 2023-08-25T02:14:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:08.122 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-08-25T02:14:08.217 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-08-25T02:14:08.218 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2023-08-25T02:14:08.218 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-08-25T02:14:08.218 INFO:teuthology.orchestra.run.smithi122.stdout:Installing group/module packages: 2023-08-25T02:14:08.218 INFO:teuthology.orchestra.run.smithi122.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-08-25T02:14:08.218 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-08-25T02:14:08.219 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-08-25T02:14:08.220 INFO:teuthology.orchestra.run.smithi122.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-08-25T02:14:08.220 INFO:teuthology.orchestra.run.smithi122.stdout:Installing module profiles: 2023-08-25T02:14:08.236 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools/common 2023-08-25T02:14:08.236 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:08.236 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2023-08-25T02:14:08.237 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools rhel8 2023-08-25T02:14:08.237 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:08.237 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-08-25T02:14:08.238 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-08-25T02:14:08.238 INFO:teuthology.orchestra.run.smithi122.stdout:Install 10 Packages 2023-08-25T02:14:08.238 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:08.238 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 18 M 2023-08-25T02:14:08.238 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 65 M 2023-08-25T02:14:08.239 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2023-08-25T02:14:08.304 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-08-25T02:14:08.379 INFO:teuthology.orchestra.run.smithi122.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 601 kB/s | 94 kB 00:00 2023-08-25T02:14:08.404 INFO:teuthology.orchestra.run.smithi122.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 225 kB/s | 41 kB 00:00 2023-08-25T02:14:08.425 INFO:teuthology.orchestra.run.smithi122.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 120 kB/s | 25 kB 00:00 2023-08-25T02:14:08.468 INFO:teuthology.orchestra.run.smithi122.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 588 kB/s | 49 kB 00:00 2023-08-25T02:14:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-08-25T02:14:08.539 INFO:teuthology.orchestra.run.smithi122.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-08-25T02:14:08.582 INFO:teuthology.orchestra.run.smithi122.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-08-25T02:14:08.725 INFO:teuthology.orchestra.run.smithi122.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 26 MB/s | 8.0 MB 00:00 2023-08-25T02:14:08.750 INFO:teuthology.orchestra.run.smithi122.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 980 kB/s | 213 kB 00:00 2023-08-25T02:14:08.837 INFO:teuthology.orchestra.run.smithi122.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 20 MB/s | 2.2 MB 00:00 2023-08-25T02:14:08.884 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-08-25T02:14:08.933 INFO:teuthology.orchestra.run.smithi122.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-08-25T02:14:08.934 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:14:08.934 INFO:teuthology.orchestra.run.smithi122.stdout:Total 25 MB/s | 18 MB 00:00 2023-08-25T02:14:09.136 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2023-08-25T02:14:09.153 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2023-08-25T02:14:09.153 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2023-08-25T02:14:09.345 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2023-08-25T02:14:09.345 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2023-08-25T02:14:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-08-25T02:14:09.680 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-08-25T02:14:09.772 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================================ 2023-08-25T02:14:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================================ 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout:Installing group/module packages: 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-08-25T02:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-08-25T02:14:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout:Installing module profiles: 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools/common 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools rhel8 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:09.775 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-08-25T02:14:09.776 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================================ 2023-08-25T02:14:09.776 INFO:teuthology.orchestra.run.smithi121.stdout:Install 10 Packages 2023-08-25T02:14:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:09.782 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 18 M 2023-08-25T02:14:09.782 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 65 M 2023-08-25T02:14:09.783 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-08-25T02:14:09.882 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2023-08-25T02:14:09.947 INFO:teuthology.orchestra.run.smithi121.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 147 kB/s | 25 kB 00:00 2023-08-25T02:14:09.972 INFO:teuthology.orchestra.run.smithi121.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 483 kB/s | 94 kB 00:00 2023-08-25T02:14:09.995 INFO:teuthology.orchestra.run.smithi121.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 186 kB/s | 41 kB 00:00 2023-08-25T02:14:10.062 INFO:teuthology.orchestra.run.smithi121.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 421 kB/s | 49 kB 00:00 2023-08-25T02:14:10.098 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-08-25T02:14:10.187 INFO:teuthology.orchestra.run.smithi121.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.4 MB/s | 524 kB 00:00 2023-08-25T02:14:10.237 INFO:teuthology.orchestra.run.smithi121.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 853 kB/s | 149 kB 00:00 2023-08-25T02:14:10.272 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:10.279 INFO:teuthology.orchestra.run.smithi121.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.2 MB/s | 213 kB 00:00 2023-08-25T02:14:10.431 INFO:teuthology.orchestra.run.smithi121.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 18 MB/s | 8.0 MB 00:00 2023-08-25T02:14:10.491 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-08-25T02:14:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 9.7 MB/s | 2.2 MB 00:00 2023-08-25T02:14:10.505 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-08-25T02:14:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 16 MB/s | 7.0 MB 00:00 2023-08-25T02:14:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:14:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 18 MB 00:00 2023-08-25T02:14:10.737 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-08-25T02:14:10.754 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-08-25T02:14:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-08-25T02:14:10.885 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-08-25T02:14:10.886 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-08-25T02:14:10.929 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-08-25T02:14:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-08-25T02:14:11.068 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-08-25T02:14:11.090 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:11.092 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-08-25T02:14:11.343 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:11.343 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-08-25T02:14:11.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-08-25T02:14:11.345 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-08-25T02:14:11.604 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-08-25T02:14:11.794 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-08-25T02:14:11.860 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-08-25T02:14:11.980 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:12.152 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-08-25T02:14:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-08-25T02:14:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-08-25T02:14:12.543 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:43 ago on Fri 25 Aug 2023 02:11:29 AM UTC. 2023-08-25T02:14:12.711 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-08-25T02:14:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-08-25T02:14:13.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-08-25T02:14:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-08-25T02:14:13.484 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:13.488 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-08-25T02:14:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-08-25T02:14:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-08-25T02:14:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-08-25T02:14:14.385 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-08-25T02:14:14.621 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-08-25T02:14:14.832 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-08-25T02:14:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-08-25T02:14:14.869 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:27 ago on Fri 25 Aug 2023 02:11:47 AM UTC. 2023-08-25T02:14:15.047 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:15.315 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-08-25T02:14:15.316 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-08-25T02:14:15.316 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-08-25T02:14:15.316 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-08-25T02:14:15.316 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-08-25T02:14:15.317 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-08-25T02:14:15.317 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-08-25T02:14:15.317 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-08-25T02:14:15.317 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-08-25T02:14:15.317 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-08-25T02:14:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-08-25T02:14:16.437 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:37 ago on Fri 25 Aug 2023 02:11:38 AM UTC. 2023-08-25T02:14:22.698 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2023-08-25T02:14:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:22.902 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2023-08-25T02:14:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: yajl-2.1.0-10.el8.x86_64 2023-08-25T02:14:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:22.904 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-08-25T02:14:23.035 DEBUG:teuthology.parallel:result is None 2023-08-25T02:14:23.799 INFO:teuthology.orchestra.run.smithi089.stdout:Installed products updated. 2023-08-25T02:14:23.993 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-08-25T02:14:23.994 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout: yajl-2.1.0-10.el8.x86_64 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:23.995 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-08-25T02:14:24.156 DEBUG:teuthology.parallel:result is None 2023-08-25T02:14:24.588 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-08-25T02:14:24.790 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:24.790 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-08-25T02:14:24.798 INFO:teuthology.orchestra.run.smithi121.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:24.798 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:24.798 INFO:teuthology.orchestra.run.smithi121.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:24.798 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-08-25T02:14:24.798 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-08-25T02:14:24.799 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-08-25T02:14:24.799 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:24.799 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-08-25T02:14:24.799 INFO:teuthology.orchestra.run.smithi121.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-08-25T02:14:24.799 INFO:teuthology.orchestra.run.smithi121.stdout: yajl-2.1.0-10.el8.x86_64 2023-08-25T02:14:24.802 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:24.802 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-08-25T02:14:24.956 DEBUG:teuthology.parallel:result is None 2023-08-25T02:14:24.957 INFO:teuthology.run_tasks:Running task install... 2023-08-25T02:14:24.989 DEBUG:teuthology.task.install:project ceph 2023-08-25T02:14:24.996 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'}} 2023-08-25T02:14:24.996 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'} 2023-08-25T02:14:24.996 INFO:teuthology.task.install:Using flavor: default 2023-08-25T02:14:25.013 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-08-25T02:14:25.014 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-08-25T02:14:25.014 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'tag': None, 'wait_for_package': False} 2023-08-25T02:14:25.015 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:14:25.017 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'tag': None, 'wait_for_package': False} 2023-08-25T02:14:25.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:14:25.019 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'tag': None, 'wait_for_package': False} 2023-08-25T02:14:25.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:14:25.228 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/ 2023-08-25T02:14:25.229 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5703.gba6b39e4 2023-08-25T02:14:25.253 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/ 2023-08-25T02:14:25.253 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5703.gba6b39e4 2023-08-25T02:14:25.306 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/ 2023-08-25T02:14:25.306 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5703.gba6b39e4 2023-08-25T02:14:25.377 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-25T02:14:25.377 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:14:25.377 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-25T02:14:25.400 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-25T02:14:25.400 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:14:25.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-25T02:14:25.436 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-08-25T02:14:25.437 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba6b39e4f4f77fb95c747099b53db0467d435f56/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-25T02:14:25.438 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-08-25T02:14:25.438 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba6b39e4f4f77fb95c747099b53db0467d435f56/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-25T02:14:25.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-25T02:14:25.525 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-25T02:14:25.526 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:14:25.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-25T02:14:25.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-25T02:14:25.567 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-08-25T02:14:25.567 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba6b39e4f4f77fb95c747099b53db0467d435f56/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-25T02:14:25.597 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-25T02:14:25.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-25T02:14:25.656 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-25T02:14:25.679 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-08-25T02:14:25.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-08-25T02:14:25.758 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2023-08-25T02:14:25.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2023-08-25T02:14:25.764 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-25T02:14:25.892 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2023-08-25T02:14:25.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2023-08-25T02:14:26.061 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:26.139 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:26.298 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:27.106 INFO:teuthology.orchestra.run.smithi121.stdout:44 files removed 2023-08-25T02:14:27.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-08-25T02:14:27.402 INFO:teuthology.orchestra.run.smithi089.stdout:44 files removed 2023-08-25T02:14:27.402 INFO:teuthology.orchestra.run.smithi122.stdout:44 files removed 2023-08-25T02:14:27.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-08-25T02:14:27.457 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-08-25T02:14:27.508 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:27.787 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:27.809 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-08-25T02:14:29.039 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 206 kB/s | 81 kB 00:00 2023-08-25T02:14:29.187 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 177 kB/s | 81 kB 00:00 2023-08-25T02:14:29.390 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-08-25T02:14:29.457 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-08-25T02:14:29.472 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 132 kB/s | 81 kB 00:00 2023-08-25T02:14:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2023-08-25T02:14:29.730 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 42 kB/s | 9.4 kB 00:00 2023-08-25T02:14:29.774 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2023-08-25T02:14:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-08-25T02:14:29.917 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-08-25T02:14:30.048 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2023-08-25T02:14:30.198 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-08-25T02:14:30.935 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-08-25T02:14:32.452 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2023-08-25T02:14:34.369 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 548 kB/s | 24 kB 00:00 2023-08-25T02:14:35.176 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 2.9 MB/s | 16 MB 00:05 2023-08-25T02:14:35.654 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 MB/s | 53 MB 00:01 2023-08-25T02:14:35.916 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 409 kB/s | 24 kB 00:00 2023-08-25T02:14:37.112 INFO:teuthology.orchestra.run.smithi089.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-08-25T02:14:38.649 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 278 kB/s | 24 kB 00:00 2023-08-25T02:14:39.926 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 MB/s | 47 MB 00:01 2023-08-25T02:14:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 MB/s | 47 MB 00:01 2023-08-25T02:14:45.834 INFO:teuthology.orchestra.run.smithi089.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 MB/s | 47 MB 00:01 2023-08-25T02:14:47.948 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 53 MB 00:01 2023-08-25T02:14:55.068 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:55.069 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:55.083 INFO:teuthology.orchestra.run.smithi122.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:55.088 INFO:teuthology.orchestra.run.smithi122.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:55.089 INFO:teuthology.orchestra.run.smithi122.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-08-25T02:14:55.089 INFO:teuthology.orchestra.run.smithi122.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-08-25T02:14:55.089 INFO:teuthology.orchestra.run.smithi122.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-08-25T02:14:55.092 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-08-25T02:14:55.093 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-08-25T02:14:55.223 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-08-25T02:14:55.233 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-08-25T02:14:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2023-08-25T02:14:55.234 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-08-25T02:14:55.234 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2023-08-25T02:14:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-08-25T02:14:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-08-25T02:14:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.4 k 2023-08-25T02:14:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 5.4 M 2023-08-25T02:14:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 862 k 2023-08-25T02:14:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 167 k 2023-08-25T02:14:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 1.6 M 2023-08-25T02:14:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 132 k 2023-08-25T02:14:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 1.7 M 2023-08-25T02:14:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 7.4 M 2023-08-25T02:14:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 47 k 2023-08-25T02:14:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 14 M 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 48 M 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 261 k 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 171 k 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 260 k 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 18 k 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-08-25T02:14:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 31 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 784 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-08-25T02:14:55.259 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-08-25T02:14:55.260 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-08-25T02:14:55.260 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-08-25T02:14:55.261 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-08-25T02:14:55.261 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 197 k 2023-08-25T02:14:55.262 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 371 k 2023-08-25T02:14:55.262 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 364 k 2023-08-25T02:14:55.262 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 107 k 2023-08-25T02:14:55.262 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 94 k 2023-08-25T02:14:55.265 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.5 M 2023-08-25T02:14:55.265 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 179 k 2023-08-25T02:14:55.266 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-08-25T02:14:55.267 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2023-08-25T02:14:55.267 INFO:teuthology.orchestra.run.smithi122.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-08-25T02:14:55.267 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.7 M 2023-08-25T02:14:55.267 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.6 M 2023-08-25T02:14:55.267 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 22 M 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 23 k 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 2.3 M 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 242 k 2023-08-25T02:14:55.268 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 4.6 M 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 18 M 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 15 k 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 24 k 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-25T02:14:55.269 INFO:teuthology.orchestra.run.smithi122.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-08-25T02:14:55.276 INFO:teuthology.orchestra.run.smithi122.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-08-25T02:14:55.277 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:55.277 INFO:teuthology.orchestra.run.smithi122.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-08-25T02:14:55.277 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-25T02:14:55.277 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-25T02:14:55.277 INFO:teuthology.orchestra.run.smithi122.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-08-25T02:14:55.285 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-25T02:14:55.285 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-25T02:14:55.286 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 473 k 2023-08-25T02:14:55.288 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-25T02:14:55.288 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.8 M 2023-08-25T02:14:55.290 INFO:teuthology.orchestra.run.smithi122.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-08-25T02:14:55.290 INFO:teuthology.orchestra.run.smithi122.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-08-25T02:14:55.290 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-25T02:14:55.290 INFO:teuthology.orchestra.run.smithi122.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-08-25T02:14:55.290 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-25T02:14:55.291 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-25T02:14:55.291 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-25T02:14:55.291 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-25T02:14:55.291 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-08-25T02:14:55.291 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-08-25T02:14:55.292 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-08-25T02:14:55.292 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-25T02:14:55.292 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-25T02:14:55.292 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-25T02:14:55.292 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-25T02:14:55.293 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-25T02:14:55.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 45 k 2023-08-25T02:14:55.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 126 k 2023-08-25T02:14:55.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-25T02:14:55.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-25T02:14:55.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-25T02:14:55.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-25T02:14:55.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-25T02:14:55.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-25T02:14:55.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-25T02:14:55.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-25T02:14:55.303 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-25T02:14:55.306 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-25T02:14:55.306 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-25T02:14:55.306 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:55.306 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-25T02:14:55.306 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-25T02:14:55.307 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-25T02:14:55.307 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-25T02:14:55.318 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-25T02:14:55.320 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-25T02:14:55.320 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-25T02:14:55.320 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-25T02:14:55.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-25T02:14:55.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-25T02:14:55.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-25T02:14:55.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-25T02:14:55.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-25T02:14:55.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-25T02:14:55.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-25T02:14:55.323 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-25T02:14:55.323 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-25T02:14:55.323 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-25T02:14:55.324 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:55.324 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-25T02:14:55.324 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-25T02:14:55.325 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-25T02:14:55.325 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-25T02:14:55.325 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-25T02:14:55.325 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-25T02:14:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-25T02:14:55.328 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-25T02:14:55.328 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-25T02:14:55.328 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-25T02:14:55.329 INFO:teuthology.orchestra.run.smithi122.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-08-25T02:14:55.329 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-25T02:14:55.329 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-08-25T02:14:55.330 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2023-08-25T02:14:55.330 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql 10 2023-08-25T02:14:55.330 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:55.330 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-08-25T02:14:55.330 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-08-25T02:14:55.331 INFO:teuthology.orchestra.run.smithi122.stdout:Install 128 Packages 2023-08-25T02:14:55.331 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 3 Packages 2023-08-25T02:14:55.331 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:14:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 206 M 2023-08-25T02:14:55.332 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2023-08-25T02:14:55.429 INFO:teuthology.orchestra.run.smithi122.stdout:(1/131): ceph-18.0.0-5703.gba6b39e4.el8.x86_64. 43 kB/s | 6.4 kB 00:00 2023-08-25T02:14:55.607 INFO:teuthology.orchestra.run.smithi122.stdout:(2/131): ceph-fuse-18.0.0-5703.gba6b39e4.el8.x8 4.7 MB/s | 862 kB 00:00 2023-08-25T02:14:55.666 INFO:teuthology.orchestra.run.smithi122.stdout:(3/131): ceph-immutable-object-cache-18.0.0-570 2.8 MB/s | 167 kB 00:00 2023-08-25T02:14:55.910 INFO:teuthology.orchestra.run.smithi122.stdout:(4/131): ceph-mds-18.0.0-5703.gba6b39e4.el8.x86 9.5 MB/s | 2.3 MB 00:00 2023-08-25T02:14:56.117 INFO:teuthology.orchestra.run.smithi122.stdout:(5/131): ceph-mgr-18.0.0-5703.gba6b39e4.el8.x86 7.5 MB/s | 1.6 MB 00:00 2023-08-25T02:14:56.192 INFO:teuthology.orchestra.run.smithi122.stdout:(6/131): ceph-base-18.0.0-5703.gba6b39e4.el8.x8 5.9 MB/s | 5.4 MB 00:00 2023-08-25T02:14:56.626 INFO:teuthology.orchestra.run.smithi122.stdout:(7/131): ceph-mon-18.0.0-5703.gba6b39e4.el8.x86 9.1 MB/s | 4.6 MB 00:00 2023-08-25T02:14:56.767 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:56.768 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:56.783 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:56.784 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-08-25T02:14:56.786 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-08-25T02:14:56.788 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-08-25T02:14:56.790 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-08-25T02:14:56.792 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-08-25T02:14:56.922 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-08-25T02:14:56.935 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-08-25T02:14:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-08-25T02:14:56.935 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-08-25T02:14:56.935 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-08-25T02:14:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.4 k 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 5.4 M 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 862 k 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 167 k 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 1.6 M 2023-08-25T02:14:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 132 k 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 1.7 M 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 7.4 M 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 47 k 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 14 M 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 48 M 2023-08-25T02:14:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 261 k 2023-08-25T02:14:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 171 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 260 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 18 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-08-25T02:14:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 31 k 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 784 k 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-08-25T02:14:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 197 k 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 371 k 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 364 k 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 107 k 2023-08-25T02:14:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 94 k 2023-08-25T02:14:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.5 M 2023-08-25T02:14:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 179 k 2023-08-25T02:14:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-08-25T02:14:56.944 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2023-08-25T02:14:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-08-25T02:14:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.7 M 2023-08-25T02:14:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.6 M 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 22 M 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 23 k 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 2.3 M 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 242 k 2023-08-25T02:14:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 4.6 M 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 18 M 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 15 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 24 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-08-25T02:14:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-25T02:14:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-25T02:14:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 473 k 2023-08-25T02:14:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-25T02:14:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.8 M 2023-08-25T02:14:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-08-25T02:14:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-08-25T02:14:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-25T02:14:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-08-25T02:14:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-25T02:14:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-25T02:14:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-25T02:14:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-25T02:14:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-08-25T02:14:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-08-25T02:14:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 45 k 2023-08-25T02:14:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 126 k 2023-08-25T02:14:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-25T02:14:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-25T02:14:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-25T02:14:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-25T02:14:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-25T02:14:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-25T02:14:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-25T02:14:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-25T02:14:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-25T02:14:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-25T02:14:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-25T02:14:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-25T02:14:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-25T02:14:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-25T02:14:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-25T02:14:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-25T02:14:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-25T02:14:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-08-25T02:14:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-25T02:14:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-08-25T02:14:56.961 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling module streams: 2023-08-25T02:14:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql 10 2023-08-25T02:14:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:56.961 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-08-25T02:14:56.962 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-08-25T02:14:56.962 INFO:teuthology.orchestra.run.smithi089.stdout:Install 128 Packages 2023-08-25T02:14:56.962 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 3 Packages 2023-08-25T02:14:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:14:56.964 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 206 M 2023-08-25T02:14:56.964 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-08-25T02:14:57.454 INFO:teuthology.orchestra.run.smithi089.stdout:(1/131): ceph-18.0.0-5703.gba6b39e4.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2023-08-25T02:14:58.137 INFO:teuthology.orchestra.run.smithi089.stdout:(2/131): ceph-fuse-18.0.0-5703.gba6b39e4.el8.x8 1.2 MB/s | 862 kB 00:00 2023-08-25T02:14:58.168 INFO:teuthology.orchestra.run.smithi122.stdout:(8/131): ceph-radosgw-18.0.0-5703.gba6b39e4.el8 8.8 MB/s | 14 MB 00:01 2023-08-25T02:14:58.212 INFO:teuthology.orchestra.run.smithi089.stdout:(3/131): ceph-immutable-object-cache-18.0.0-570 2.2 MB/s | 167 kB 00:00 2023-08-25T02:14:58.346 INFO:teuthology.orchestra.run.smithi089.stdout:(4/131): ceph-base-18.0.0-5703.gba6b39e4.el8.x8 4.4 MB/s | 5.4 MB 00:01 2023-08-25T02:14:58.376 INFO:teuthology.orchestra.run.smithi122.stdout:(9/131): ceph-common-18.0.0-5703.gba6b39e4.el8. 7.2 MB/s | 22 MB 00:03 2023-08-25T02:14:58.402 INFO:teuthology.orchestra.run.smithi122.stdout:(10/131): ceph-selinux-18.0.0-5703.gba6b39e4.el 104 kB/s | 24 kB 00:00 2023-08-25T02:14:58.505 INFO:teuthology.orchestra.run.smithi089.stdout:(5/131): ceph-mgr-18.0.0-5703.gba6b39e4.el8.x86 9.9 MB/s | 1.6 MB 00:00 2023-08-25T02:14:58.588 INFO:teuthology.orchestra.run.smithi122.stdout:(11/131): ceph-osd-18.0.0-5703.gba6b39e4.el8.x8 7.4 MB/s | 18 MB 00:02 2023-08-25T02:14:58.612 INFO:teuthology.orchestra.run.smithi122.stdout:(12/131): cephfs-mirror-18.0.0-5703.gba6b39e4.e 1.2 MB/s | 260 kB 00:00 2023-08-25T02:14:58.677 INFO:teuthology.orchestra.run.smithi122.stdout:(13/131): libcephfs-devel-18.0.0-5703.gba6b39e4 339 kB/s | 31 kB 00:00 2023-08-25T02:14:58.772 INFO:teuthology.orchestra.run.smithi089.stdout:(6/131): ceph-mds-18.0.0-5703.gba6b39e4.el8.x86 4.1 MB/s | 2.3 MB 00:00 2023-08-25T02:14:58.823 INFO:teuthology.orchestra.run.smithi122.stdout:(14/131): libcephfs2-18.0.0-5703.gba6b39e4.el8. 3.6 MB/s | 784 kB 00:00 2023-08-25T02:14:58.855 INFO:teuthology.orchestra.run.smithi122.stdout:(15/131): libcephsqlite-18.0.0-5703.gba6b39e4.e 792 kB/s | 139 kB 00:00 2023-08-25T02:14:58.915 INFO:teuthology.orchestra.run.smithi122.stdout:(16/131): librados-devel-18.0.0-5703.gba6b39e4. 1.5 MB/s | 139 kB 00:00 2023-08-25T02:14:58.984 INFO:teuthology.orchestra.run.smithi089.stdout:(7/131): ceph-mon-18.0.0-5703.gba6b39e4.el8.x86 9.7 MB/s | 4.6 MB 00:00 2023-08-25T02:14:59.054 INFO:teuthology.orchestra.run.smithi122.stdout:(17/131): libradosstriper1-18.0.0-5703.gba6b39e 2.3 MB/s | 473 kB 00:00 2023-08-25T02:14:59.085 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:59.086 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-08-25T02:14:59.099 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:59.100 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-08-25T02:14:59.105 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-08-25T02:14:59.105 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-08-25T02:14:59.105 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-08-25T02:14:59.108 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-08-25T02:14:59.116 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-08-25T02:14:59.245 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-08-25T02:14:59.257 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-08-25T02:14:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-08-25T02:14:59.258 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-08-25T02:14:59.258 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-08-25T02:14:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-08-25T02:14:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-08-25T02:14:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.4 k 2023-08-25T02:14:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 5.4 M 2023-08-25T02:14:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 862 k 2023-08-25T02:14:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 167 k 2023-08-25T02:14:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 1.6 M 2023-08-25T02:14:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 132 k 2023-08-25T02:14:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 1.7 M 2023-08-25T02:14:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 7.4 M 2023-08-25T02:14:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 47 k 2023-08-25T02:14:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 14 M 2023-08-25T02:14:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 48 M 2023-08-25T02:14:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 261 k 2023-08-25T02:14:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 171 k 2023-08-25T02:14:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 260 k 2023-08-25T02:14:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 18 k 2023-08-25T02:14:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-08-25T02:14:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-08-25T02:14:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-08-25T02:14:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 31 k 2023-08-25T02:14:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 784 k 2023-08-25T02:14:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-08-25T02:14:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-08-25T02:14:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-08-25T02:14:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-08-25T02:14:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-08-25T02:14:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-08-25T02:14:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 197 k 2023-08-25T02:14:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 371 k 2023-08-25T02:14:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 364 k 2023-08-25T02:14:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 107 k 2023-08-25T02:14:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 94 k 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.5 M 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 179 k 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-08-25T02:14:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.7 M 2023-08-25T02:14:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 3.6 M 2023-08-25T02:14:59.273 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-08-25T02:14:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-08-25T02:14:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 22 M 2023-08-25T02:14:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 23 k 2023-08-25T02:14:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 2.3 M 2023-08-25T02:14:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 242 k 2023-08-25T02:14:59.276 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 4.6 M 2023-08-25T02:14:59.276 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 18 M 2023-08-25T02:14:59.278 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5703.gba6b39e4.el8 ceph-noarch 15 k 2023-08-25T02:14:59.280 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 24 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-08-25T02:14:59.282 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 139 k 2023-08-25T02:14:59.292 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-08-25T02:14:59.292 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-25T02:14:59.292 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 473 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 6.8 M 2023-08-25T02:14:59.293 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-08-25T02:14:59.294 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-08-25T02:14:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-25T02:14:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-08-25T02:14:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-08-25T02:14:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-08-25T02:14:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 45 k 2023-08-25T02:14:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el8 ceph 126 k 2023-08-25T02:14:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-25T02:14:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-25T02:14:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-25T02:14:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-25T02:14:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-25T02:14:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-25T02:14:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-25T02:14:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-25T02:14:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-25T02:14:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-25T02:14:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-25T02:14:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-25T02:14:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-25T02:14:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-25T02:14:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-25T02:14:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-25T02:14:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-25T02:14:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-25T02:14:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-25T02:14:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-25T02:14:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-25T02:14:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-25T02:14:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-25T02:14:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-25T02:14:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-25T02:14:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-25T02:14:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-25T02:14:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-25T02:14:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql 10 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-08-25T02:14:59.319 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-08-25T02:14:59.320 INFO:teuthology.orchestra.run.smithi121.stdout:Install 128 Packages 2023-08-25T02:14:59.320 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 3 Packages 2023-08-25T02:14:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:14:59.321 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 206 M 2023-08-25T02:14:59.321 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-08-25T02:14:59.753 INFO:teuthology.orchestra.run.smithi121.stdout:(1/131): ceph-18.0.0-5703.gba6b39e4.el8.x86_64. 19 kB/s | 6.4 kB 00:00 2023-08-25T02:14:59.838 INFO:teuthology.orchestra.run.smithi122.stdout:(18/131): python3-ceph-argparse-18.0.0-5703.gba 58 kB/s | 45 kB 00:00 2023-08-25T02:14:59.964 INFO:teuthology.orchestra.run.smithi122.stdout:(19/131): librgw2-18.0.0-5703.gba6b39e4.el8.x86 6.5 MB/s | 6.8 MB 00:01 2023-08-25T02:14:59.994 INFO:teuthology.orchestra.run.smithi122.stdout:(20/131): python3-ceph-common-18.0.0-5703.gba6b 792 kB/s | 126 kB 00:00 2023-08-25T02:15:00.064 INFO:teuthology.orchestra.run.smithi122.stdout:(21/131): python3-cephfs-18.0.0-5703.gba6b39e4. 1.9 MB/s | 197 kB 00:00 2023-08-25T02:15:00.384 INFO:teuthology.orchestra.run.smithi121.stdout:(2/131): ceph-fuse-18.0.0-5703.gba6b39e4.el8.x8 1.3 MB/s | 862 kB 00:00 2023-08-25T02:15:00.481 INFO:teuthology.orchestra.run.smithi122.stdout:(22/131): python3-rados-18.0.0-5703.gba6b39e4.e 755 kB/s | 371 kB 00:00 2023-08-25T02:15:00.494 INFO:teuthology.orchestra.run.smithi121.stdout:(3/131): ceph-immutable-object-cache-18.0.0-570 1.5 MB/s | 167 kB 00:00 2023-08-25T02:15:00.573 INFO:teuthology.orchestra.run.smithi122.stdout:(23/131): python3-rgw-18.0.0-5703.gba6b39e4.el8 1.1 MB/s | 107 kB 00:00 2023-08-25T02:15:00.690 INFO:teuthology.orchestra.run.smithi122.stdout:(24/131): rbd-fuse-18.0.0-5703.gba6b39e4.el8.x8 802 kB/s | 94 kB 00:00 2023-08-25T02:15:00.915 INFO:teuthology.orchestra.run.smithi122.stdout:(25/131): python3-rbd-18.0.0-5703.gba6b39e4.el8 428 kB/s | 364 kB 00:00 2023-08-25T02:15:01.220 INFO:teuthology.orchestra.run.smithi121.stdout:(4/131): ceph-base-18.0.0-5703.gba6b39e4.el8.x8 3.0 MB/s | 5.4 MB 00:01 2023-08-25T02:15:01.480 INFO:teuthology.orchestra.run.smithi089.stdout:(8/131): ceph-radosgw-18.0.0-5703.gba6b39e4.el8 5.4 MB/s | 14 MB 00:02 2023-08-25T02:15:01.573 INFO:teuthology.orchestra.run.smithi089.stdout:(9/131): ceph-selinux-18.0.0-5703.gba6b39e4.el8 266 kB/s | 24 kB 00:00 2023-08-25T02:15:01.683 INFO:teuthology.orchestra.run.smithi122.stdout:(26/131): rbd-nbd-18.0.0-5703.gba6b39e4.el8.x86 233 kB/s | 179 kB 00:00 2023-08-25T02:15:01.690 INFO:teuthology.orchestra.run.smithi121.stdout:(5/131): ceph-mgr-18.0.0-5703.gba6b39e4.el8.x86 3.4 MB/s | 1.6 MB 00:00 2023-08-25T02:15:01.770 INFO:teuthology.orchestra.run.smithi121.stdout:(6/131): ceph-mds-18.0.0-5703.gba6b39e4.el8.x86 1.8 MB/s | 2.3 MB 00:01 2023-08-25T02:15:01.805 INFO:teuthology.orchestra.run.smithi122.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5703.g 190 kB/s | 23 kB 00:00 2023-08-25T02:15:02.091 INFO:teuthology.orchestra.run.smithi122.stdout:(28/131): rbd-mirror-18.0.0-5703.gba6b39e4.el8. 2.5 MB/s | 3.5 MB 00:01 2023-08-25T02:15:02.496 INFO:teuthology.orchestra.run.smithi122.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5703.gba6b39e 192 kB/s | 132 kB 00:00 2023-08-25T02:15:02.959 INFO:teuthology.orchestra.run.smithi122.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5703.gba6b3 2.0 MB/s | 1.7 MB 00:00 2023-08-25T02:15:03.159 INFO:teuthology.orchestra.run.smithi122.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5703.gba 1.2 MB/s | 242 kB 00:00 2023-08-25T02:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout:(7/131): ceph-mon-18.0.0-5703.gba6b39e4.el8.x86 3.1 MB/s | 4.6 MB 00:01 2023-08-25T02:15:03.261 INFO:teuthology.orchestra.run.smithi122.stdout:(32/131): ceph-mgr-rook-18.0.0-5703.gba6b39e4.e 475 kB/s | 47 kB 00:00 2023-08-25T02:15:03.377 INFO:teuthology.orchestra.run.smithi122.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5703.gb 125 kB/s | 15 kB 00:00 2023-08-25T02:15:03.578 INFO:teuthology.orchestra.run.smithi122.stdout:(34/131): ceph-volume-18.0.0-5703.gba6b39e4.el8 1.3 MB/s | 261 kB 00:00 2023-08-25T02:15:03.686 INFO:teuthology.orchestra.run.smithi122.stdout:(35/131): cephadm-18.0.0-5703.gba6b39e4.el8.noa 1.5 MB/s | 171 kB 00:00 2023-08-25T02:15:03.812 INFO:teuthology.orchestra.run.smithi122.stdout:(36/131): cephfs-top-18.0.0-5703.gba6b39e4.el8. 144 kB/s | 18 kB 00:00 2023-08-25T02:15:04.269 INFO:teuthology.orchestra.run.smithi122.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 681 kB/s | 306 kB 00:00 2023-08-25T02:15:04.373 INFO:teuthology.orchestra.run.smithi122.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 543 kB/s | 59 kB 00:00 2023-08-25T02:15:04.478 INFO:teuthology.orchestra.run.smithi122.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 692 kB/s | 75 kB 00:00 2023-08-25T02:15:04.666 INFO:teuthology.orchestra.run.smithi122.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2023-08-25T02:15:04.788 INFO:teuthology.orchestra.run.smithi122.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 348 kB/s | 44 kB 00:00 2023-08-25T02:15:04.832 INFO:teuthology.orchestra.run.smithi089.stdout:(10/131): ceph-osd-18.0.0-5703.gba6b39e4.el8.x8 2.9 MB/s | 18 MB 00:06 2023-08-25T02:15:04.890 INFO:teuthology.orchestra.run.smithi122.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-08-25T02:15:04.973 INFO:teuthology.orchestra.run.smithi089.stdout:(11/131): cephfs-mirror-18.0.0-5703.gba6b39e4.e 1.8 MB/s | 260 kB 00:00 2023-08-25T02:15:04.981 INFO:teuthology.orchestra.run.smithi122.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 360 kB/s | 33 kB 00:00 2023-08-25T02:15:05.031 INFO:teuthology.orchestra.run.smithi089.stdout:(12/131): libcephfs-devel-18.0.0-5703.gba6b39e4 535 kB/s | 31 kB 00:00 2023-08-25T02:15:05.071 INFO:teuthology.orchestra.run.smithi121.stdout:(8/131): ceph-common-18.0.0-5703.gba6b39e4.el8. 3.9 MB/s | 22 MB 00:05 2023-08-25T02:15:05.097 INFO:teuthology.orchestra.run.smithi122.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 140 kB/s | 16 kB 00:00 2023-08-25T02:15:05.145 INFO:teuthology.orchestra.run.smithi121.stdout:(9/131): ceph-selinux-18.0.0-5703.gba6b39e4.el8 325 kB/s | 24 kB 00:00 2023-08-25T02:15:05.201 INFO:teuthology.orchestra.run.smithi122.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2023-08-25T02:15:05.381 INFO:teuthology.orchestra.run.smithi122.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2023-08-25T02:15:05.401 INFO:teuthology.orchestra.run.smithi089.stdout:(13/131): libcephfs2-18.0.0-5703.gba6b39e4.el8. 2.1 MB/s | 784 kB 00:00 2023-08-25T02:15:05.485 INFO:teuthology.orchestra.run.smithi122.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 813 kB/s | 81 kB 00:00 2023-08-25T02:15:05.499 INFO:teuthology.orchestra.run.smithi089.stdout:(14/131): libcephsqlite-18.0.0-5703.gba6b39e4.e 1.4 MB/s | 139 kB 00:00 2023-08-25T02:15:05.574 INFO:teuthology.orchestra.run.smithi122.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-08-25T02:15:05.599 INFO:teuthology.orchestra.run.smithi089.stdout:(15/131): librados-devel-18.0.0-5703.gba6b39e4. 1.4 MB/s | 139 kB 00:00 2023-08-25T02:15:05.657 INFO:teuthology.orchestra.run.smithi122.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 211 kB/s | 18 kB 00:00 2023-08-25T02:15:05.859 INFO:teuthology.orchestra.run.smithi089.stdout:(16/131): libradosstriper1-18.0.0-5703.gba6b39e 1.8 MB/s | 473 kB 00:00 2023-08-25T02:15:05.974 INFO:teuthology.orchestra.run.smithi122.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 3.7 MB/s | 1.2 MB 00:00 2023-08-25T02:15:06.107 INFO:teuthology.orchestra.run.smithi122.stdout:(51/131): ceph-mgr-diskprediction-local-18.0.0- 2.1 MB/s | 7.4 MB 00:03 2023-08-25T02:15:06.117 INFO:teuthology.orchestra.run.smithi122.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 343 kB/s | 49 kB 00:00 2023-08-25T02:15:06.217 INFO:teuthology.orchestra.run.smithi122.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 603 kB/s | 60 kB 00:00 2023-08-25T02:15:06.329 INFO:teuthology.orchestra.run.smithi122.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-08-25T02:15:06.427 INFO:teuthology.orchestra.run.smithi122.stdout:(55/131): python3-more-itertools-7.2.0-3.el8.no 174 kB/s | 59 kB 00:00 2023-08-25T02:15:06.454 INFO:teuthology.orchestra.run.smithi122.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2023-08-25T02:15:06.537 INFO:teuthology.orchestra.run.smithi122.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 330 kB/s | 34 kB 00:00 2023-08-25T02:15:06.627 INFO:teuthology.orchestra.run.smithi122.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 643 kB/s | 64 kB 00:00 2023-08-25T02:15:06.652 INFO:teuthology.orchestra.run.smithi122.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 979 kB/s | 196 kB 00:00 2023-08-25T02:15:06.720 INFO:teuthology.orchestra.run.smithi122.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2023-08-25T02:15:06.778 INFO:teuthology.orchestra.run.smithi122.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-08-25T02:15:06.812 INFO:teuthology.orchestra.run.smithi122.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 318 kB/s | 29 kB 00:00 2023-08-25T02:15:06.872 INFO:teuthology.orchestra.run.smithi122.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-08-25T02:15:06.933 INFO:teuthology.orchestra.run.smithi122.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2023-08-25T02:15:06.972 INFO:teuthology.orchestra.run.smithi122.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-08-25T02:15:07.033 INFO:teuthology.orchestra.run.smithi122.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 611 kB/s | 61 kB 00:00 2023-08-25T02:15:07.108 INFO:teuthology.orchestra.run.smithi122.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 639 kB/s | 85 kB 00:00 2023-08-25T02:15:07.131 INFO:teuthology.orchestra.run.smithi122.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2023-08-25T02:15:07.156 INFO:teuthology.orchestra.run.smithi122.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-08-25T02:15:07.308 INFO:teuthology.orchestra.run.smithi122.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-08-25T02:15:07.357 INFO:teuthology.orchestra.run.smithi122.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2023-08-25T02:15:07.449 INFO:teuthology.orchestra.run.smithi122.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.4 MB/s | 200 kB 00:00 2023-08-25T02:15:07.509 INFO:teuthology.orchestra.run.smithi122.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2023-08-25T02:15:07.618 INFO:teuthology.orchestra.run.smithi122.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2023-08-25T02:15:07.683 INFO:teuthology.orchestra.run.smithi122.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 274 kB/s | 48 kB 00:00 2023-08-25T02:15:07.859 INFO:teuthology.orchestra.run.smithi122.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 359 kB/s | 87 kB 00:00 2023-08-25T02:15:08.008 INFO:teuthology.orchestra.run.smithi122.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 131 kB/s | 43 kB 00:00 2023-08-25T02:15:08.036 INFO:teuthology.orchestra.run.smithi122.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 623 kB/s | 109 kB 00:00 2023-08-25T02:15:08.187 INFO:teuthology.orchestra.run.smithi122.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2023-08-25T02:15:08.209 INFO:teuthology.orchestra.run.smithi122.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.1 MB/s | 560 kB 00:00 2023-08-25T02:15:08.293 INFO:teuthology.orchestra.run.smithi122.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 172 kB/s | 19 kB 00:00 2023-08-25T02:15:08.420 INFO:teuthology.orchestra.run.smithi122.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 742 kB/s | 155 kB 00:00 2023-08-25T02:15:08.460 INFO:teuthology.orchestra.run.smithi122.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 3.3 MB/s | 559 kB 00:00 2023-08-25T02:15:08.544 INFO:teuthology.orchestra.run.smithi122.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 143 kB/s | 18 kB 00:00 2023-08-25T02:15:08.611 INFO:teuthology.orchestra.run.smithi122.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2023-08-25T02:15:08.653 INFO:teuthology.orchestra.run.smithi121.stdout:(10/131): ceph-radosgw-18.0.0-5703.gba6b39e4.el 2.5 MB/s | 14 MB 00:05 2023-08-25T02:15:08.695 INFO:teuthology.orchestra.run.smithi122.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-08-25T02:15:08.807 INFO:teuthology.orchestra.run.smithi122.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 12 MB/s | 2.3 MB 00:00 2023-08-25T02:15:08.849 INFO:teuthology.orchestra.run.smithi122.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 480 kB/s | 72 kB 00:00 2023-08-25T02:15:08.860 INFO:teuthology.orchestra.run.smithi121.stdout:(11/131): cephfs-mirror-18.0.0-5703.gba6b39e4.e 1.2 MB/s | 260 kB 00:00 2023-08-25T02:15:08.946 INFO:teuthology.orchestra.run.smithi121.stdout:(12/131): libcephfs-devel-18.0.0-5703.gba6b39e4 373 kB/s | 31 kB 00:00 2023-08-25T02:15:08.997 INFO:teuthology.orchestra.run.smithi122.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 399 kB/s | 60 kB 00:00 2023-08-25T02:15:09.022 INFO:teuthology.orchestra.run.smithi122.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 380 kB/s | 83 kB 00:00 2023-08-25T02:15:09.080 INFO:teuthology.orchestra.run.smithi122.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.1 MB/s | 98 kB 00:00 2023-08-25T02:15:09.107 INFO:teuthology.orchestra.run.smithi122.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 689 kB/s | 58 kB 00:00 2023-08-25T02:15:09.177 INFO:teuthology.orchestra.run.smithi122.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 419 kB/s | 39 kB 00:00 2023-08-25T02:15:09.276 INFO:teuthology.orchestra.run.smithi122.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.7 MB/s | 171 kB 00:00 2023-08-25T02:15:09.300 INFO:teuthology.orchestra.run.smithi122.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.3 MB/s | 644 kB 00:00 2023-08-25T02:15:09.432 INFO:teuthology.orchestra.run.smithi122.stdout:(96/131): NetworkManager-initscripts-updown-1.3 867 kB/s | 138 kB 00:00 2023-08-25T02:15:09.532 INFO:teuthology.orchestra.run.smithi122.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2023-08-25T02:15:09.561 INFO:teuthology.orchestra.run.smithi121.stdout:(13/131): libcephfs2-18.0.0-5703.gba6b39e4.el8. 1.2 MB/s | 784 kB 00:00 2023-08-25T02:15:09.720 INFO:teuthology.orchestra.run.smithi121.stdout:(14/131): libcephsqlite-18.0.0-5703.gba6b39e4.e 878 kB/s | 139 kB 00:00 2023-08-25T02:15:09.742 INFO:teuthology.orchestra.run.smithi122.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 31 MB/s | 14 MB 00:00 2023-08-25T02:15:09.768 INFO:teuthology.orchestra.run.smithi122.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 232 kB/s | 54 kB 00:00 2023-08-25T02:15:09.852 INFO:teuthology.orchestra.run.smithi122.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2023-08-25T02:15:09.880 INFO:teuthology.orchestra.run.smithi121.stdout:(15/131): librados-devel-18.0.0-5703.gba6b39e4. 878 kB/s | 139 kB 00:00 2023-08-25T02:15:09.887 INFO:teuthology.orchestra.run.smithi122.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-08-25T02:15:09.968 INFO:teuthology.orchestra.run.smithi122.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 332 kB/s | 39 kB 00:00 2023-08-25T02:15:10.047 INFO:teuthology.orchestra.run.smithi122.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 4.1 MB/s | 688 kB 00:00 2023-08-25T02:15:10.128 INFO:teuthology.orchestra.run.smithi122.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2023-08-25T02:15:10.151 INFO:teuthology.orchestra.run.smithi089.stdout:(17/131): librgw2-18.0.0-5703.gba6b39e4.el8.x86 1.6 MB/s | 6.8 MB 00:04 2023-08-25T02:15:10.151 INFO:teuthology.orchestra.run.smithi122.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2023-08-25T02:15:10.239 INFO:teuthology.orchestra.run.smithi121.stdout:(16/131): libradosstriper1-18.0.0-5703.gba6b39e 1.3 MB/s | 473 kB 00:00 2023-08-25T02:15:10.240 INFO:teuthology.orchestra.run.smithi089.stdout:(18/131): python3-ceph-argparse-18.0.0-5703.gba 488 kB/s | 45 kB 00:00 2023-08-25T02:15:10.251 INFO:teuthology.orchestra.run.smithi122.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-08-25T02:15:10.327 INFO:teuthology.orchestra.run.smithi122.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.0 MB/s | 713 kB 00:00 2023-08-25T02:15:10.418 INFO:teuthology.orchestra.run.smithi089.stdout:(19/131): python3-ceph-common-18.0.0-5703.gba6b 718 kB/s | 126 kB 00:00 2023-08-25T02:15:10.497 INFO:teuthology.orchestra.run.smithi122.stdout:(108/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-08-25T02:15:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:(17/131): ceph-osd-18.0.0-5703.gba6b39e4.el8.x8 2.0 MB/s | 18 MB 00:08 2023-08-25T02:15:10.521 INFO:teuthology.orchestra.run.smithi122.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 138 kB/s | 27 kB 00:00 2023-08-25T02:15:10.604 INFO:teuthology.orchestra.run.smithi122.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.9 MB/s | 320 kB 00:00 2023-08-25T02:15:10.621 INFO:teuthology.orchestra.run.smithi121.stdout:(18/131): python3-ceph-argparse-18.0.0-5703.gba 447 kB/s | 45 kB 00:00 2023-08-25T02:15:10.627 INFO:teuthology.orchestra.run.smithi089.stdout:(20/131): python3-cephfs-18.0.0-5703.gba6b39e4. 945 kB/s | 197 kB 00:00 2023-08-25T02:15:10.655 INFO:teuthology.orchestra.run.smithi122.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 328 kB/s | 44 kB 00:00 2023-08-25T02:15:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:(19/131): python3-ceph-common-18.0.0-5703.gba6b 1.1 MB/s | 126 kB 00:00 2023-08-25T02:15:10.807 INFO:teuthology.orchestra.run.smithi122.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2023-08-25T02:15:10.862 INFO:teuthology.orchestra.run.smithi122.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 2.1 MB/s | 457 kB 00:00 2023-08-25T02:15:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:(20/131): python3-cephfs-18.0.0-5703.gba6b39e4. 1.0 MB/s | 197 kB 00:00 2023-08-25T02:15:10.932 INFO:teuthology.orchestra.run.smithi122.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.7 MB/s | 225 kB 00:00 2023-08-25T02:15:10.991 INFO:teuthology.orchestra.run.smithi089.stdout:(21/131): python3-rados-18.0.0-5703.gba6b39e4.e 1.0 MB/s | 371 kB 00:00 2023-08-25T02:15:10.996 INFO:teuthology.orchestra.run.smithi122.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 479 kB/s | 64 kB 00:00 2023-08-25T02:15:11.066 INFO:teuthology.orchestra.run.smithi122.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 945 kB/s | 126 kB 00:00 2023-08-25T02:15:11.115 INFO:teuthology.orchestra.run.smithi122.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2023-08-25T02:15:11.177 INFO:teuthology.orchestra.run.smithi122.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 951 kB/s | 103 kB 00:00 2023-08-25T02:15:11.249 INFO:teuthology.orchestra.run.smithi122.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2023-08-25T02:15:11.374 INFO:teuthology.orchestra.run.smithi121.stdout:(21/131): python3-rados-18.0.0-5703.gba6b39e4.e 806 kB/s | 371 kB 00:00 2023-08-25T02:15:11.399 INFO:teuthology.orchestra.run.smithi122.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 22 MB/s | 4.9 MB 00:00 2023-08-25T02:15:11.419 INFO:teuthology.orchestra.run.smithi089.stdout:(22/131): python3-rbd-18.0.0-5703.gba6b39e4.el8 857 kB/s | 364 kB 00:00 2023-08-25T02:15:11.448 INFO:teuthology.orchestra.run.smithi122.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 987 kB/s | 198 kB 00:00 2023-08-25T02:15:11.628 INFO:teuthology.orchestra.run.smithi122.stdout:(122/131): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-08-25T02:15:11.629 INFO:teuthology.orchestra.run.smithi089.stdout:(23/131): python3-rgw-18.0.0-5703.gba6b39e4.el8 564 kB/s | 107 kB 00:00 2023-08-25T02:15:11.645 INFO:teuthology.orchestra.run.smithi122.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-08-25T02:15:11.764 INFO:teuthology.orchestra.run.smithi121.stdout:(22/131): python3-rbd-18.0.0-5703.gba6b39e4.el8 935 kB/s | 364 kB 00:00 2023-08-25T02:15:11.780 INFO:teuthology.orchestra.run.smithi089.stdout:(24/131): rbd-fuse-18.0.0-5703.gba6b39e4.el8.x8 554 kB/s | 94 kB 00:00 2023-08-25T02:15:11.793 INFO:teuthology.orchestra.run.smithi122.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-08-25T02:15:11.908 INFO:teuthology.orchestra.run.smithi122.stdout:(125/131): postgresql-server-10.21-2.module+el8 19 MB/s | 5.1 MB 00:00 2023-08-25T02:15:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:(23/131): python3-rgw-18.0.0-5703.gba6b39e4.el8 642 kB/s | 107 kB 00:00 2023-08-25T02:15:11.958 INFO:teuthology.orchestra.run.smithi122.stdout:(126/131): postgresql-contrib-10.21-2.module+el 4.7 MB/s | 808 kB 00:00 2023-08-25T02:15:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:(24/131): rbd-fuse-18.0.0-5703.gba6b39e4.el8.x8 1.0 MB/s | 94 kB 00:00 2023-08-25T02:15:12.125 INFO:teuthology.orchestra.run.smithi122.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 6.9 MB/s | 1.5 MB 00:00 2023-08-25T02:15:14.459 INFO:teuthology.orchestra.run.smithi122.stdout:(128/131): librados2-18.0.0-5703.gba6b39e4.el8. 1.5 MB/s | 3.7 MB 00:02 2023-08-25T02:15:14.531 INFO:teuthology.orchestra.run.smithi121.stdout:(25/131): rbd-mirror-18.0.0-5703.gba6b39e4.el8. 1.4 MB/s | 3.5 MB 00:02 2023-08-25T02:15:14.629 INFO:teuthology.orchestra.run.smithi122.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.9 MB/s | 798 kB 00:00 2023-08-25T02:15:14.730 INFO:teuthology.orchestra.run.smithi121.stdout:(26/131): rbd-nbd-18.0.0-5703.gba6b39e4.el8.x86 905 kB/s | 179 kB 00:00 2023-08-25T02:15:14.827 INFO:teuthology.orchestra.run.smithi121.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5703.g 246 kB/s | 23 kB 00:00 2023-08-25T02:15:14.992 INFO:teuthology.orchestra.run.smithi121.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5703.gba6b39e 833 kB/s | 132 kB 00:00 2023-08-25T02:15:15.393 INFO:teuthology.orchestra.run.smithi089.stdout:(25/131): rbd-mirror-18.0.0-5703.gba6b39e4.el8. 1.0 MB/s | 3.5 MB 00:03 2023-08-25T02:15:15.636 INFO:teuthology.orchestra.run.smithi089.stdout:(26/131): rbd-nbd-18.0.0-5703.gba6b39e4.el8.x86 766 kB/s | 179 kB 00:00 2023-08-25T02:15:15.723 INFO:teuthology.orchestra.run.smithi089.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5703.g 252 kB/s | 23 kB 00:00 2023-08-25T02:15:15.902 INFO:teuthology.orchestra.run.smithi089.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5703.gba6b39e 720 kB/s | 132 kB 00:00 2023-08-25T02:15:16.084 INFO:teuthology.orchestra.run.smithi121.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5703.gba6b3 1.6 MB/s | 1.7 MB 00:01 2023-08-25T02:15:16.818 INFO:teuthology.orchestra.run.smithi122.stdout:(130/131): librbd1-18.0.0-5703.gba6b39e4.el8.x8 792 kB/s | 3.6 MB 00:04 2023-08-25T02:15:17.384 INFO:teuthology.orchestra.run.smithi089.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5703.gba6b3 1.2 MB/s | 1.7 MB 00:01 2023-08-25T02:15:18.182 INFO:teuthology.orchestra.run.smithi121.stdout:(30/131): librgw2-18.0.0-5703.gba6b39e4.el8.x86 876 kB/s | 6.8 MB 00:07 2023-08-25T02:15:18.491 INFO:teuthology.orchestra.run.smithi121.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5703.gba 784 kB/s | 242 kB 00:00 2023-08-25T02:15:18.599 INFO:teuthology.orchestra.run.smithi121.stdout:(32/131): ceph-mgr-rook-18.0.0-5703.gba6b39e4.e 438 kB/s | 47 kB 00:00 2023-08-25T02:15:18.707 INFO:teuthology.orchestra.run.smithi121.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5703.gb 134 kB/s | 15 kB 00:00 2023-08-25T02:15:19.054 INFO:teuthology.orchestra.run.smithi121.stdout:(34/131): ceph-volume-18.0.0-5703.gba6b39e4.el8 845 kB/s | 261 kB 00:00 2023-08-25T02:15:19.275 INFO:teuthology.orchestra.run.smithi121.stdout:(35/131): cephadm-18.0.0-5703.gba6b39e4.el8.noa 663 kB/s | 171 kB 00:00 2023-08-25T02:15:19.386 INFO:teuthology.orchestra.run.smithi121.stdout:(36/131): cephfs-top-18.0.0-5703.gba6b39e4.el8. 166 kB/s | 18 kB 00:00 2023-08-25T02:15:19.525 INFO:teuthology.orchestra.run.smithi121.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 2.1 MB/s | 306 kB 00:00 2023-08-25T02:15:19.567 INFO:teuthology.orchestra.run.smithi121.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-08-25T02:15:19.609 INFO:teuthology.orchestra.run.smithi121.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-08-25T02:15:19.677 INFO:teuthology.orchestra.run.smithi121.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 7.0 MB/s | 476 kB 00:00 2023-08-25T02:15:19.718 INFO:teuthology.orchestra.run.smithi121.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-08-25T02:15:19.808 INFO:teuthology.orchestra.run.smithi121.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.4 MB/s | 185 kB 00:00 2023-08-25T02:15:19.858 INFO:teuthology.orchestra.run.smithi121.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 793 kB/s | 33 kB 00:00 2023-08-25T02:15:19.878 INFO:teuthology.orchestra.run.smithi121.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 389 kB/s | 16 kB 00:00 2023-08-25T02:15:19.920 INFO:teuthology.orchestra.run.smithi121.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 4.1 MB/s | 173 kB 00:00 2023-08-25T02:15:19.970 INFO:teuthology.orchestra.run.smithi121.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 7.5 MB/s | 384 kB 00:00 2023-08-25T02:15:20.003 INFO:teuthology.orchestra.run.smithi121.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 2.4 MB/s | 81 kB 00:00 2023-08-25T02:15:20.072 INFO:teuthology.orchestra.run.smithi121.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-08-25T02:15:20.112 INFO:teuthology.orchestra.run.smithi121.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 424 kB/s | 18 kB 00:00 2023-08-25T02:15:20.199 INFO:teuthology.orchestra.run.smithi121.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 14 MB/s | 1.2 MB 00:00 2023-08-25T02:15:20.220 INFO:teuthology.orchestra.run.smithi089.stdout:(30/131): ceph-common-18.0.0-5703.gba6b39e4.el8 986 kB/s | 22 MB 00:23 2023-08-25T02:15:20.233 INFO:teuthology.orchestra.run.smithi121.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.4 MB/s | 49 kB 00:00 2023-08-25T02:15:20.277 INFO:teuthology.orchestra.run.smithi121.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.4 MB/s | 59 kB 00:00 2023-08-25T02:15:20.340 INFO:teuthology.orchestra.run.smithi121.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 906 kB/s | 60 kB 00:00 2023-08-25T02:15:20.399 INFO:teuthology.orchestra.run.smithi121.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-08-25T02:15:20.431 INFO:teuthology.orchestra.run.smithi121.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 471 kB/s | 16 kB 00:00 2023-08-25T02:15:20.466 INFO:teuthology.orchestra.run.smithi121.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 1.0 MB/s | 34 kB 00:00 2023-08-25T02:15:20.513 INFO:teuthology.orchestra.run.smithi089.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5703.gba 829 kB/s | 242 kB 00:00 2023-08-25T02:15:20.517 INFO:teuthology.orchestra.run.smithi121.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-08-25T02:15:20.562 INFO:teuthology.orchestra.run.smithi121.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2023-08-25T02:15:20.639 INFO:teuthology.orchestra.run.smithi089.stdout:(32/131): ceph-mgr-rook-18.0.0-5703.gba6b39e4.e 474 kB/s | 47 kB 00:00 2023-08-25T02:15:20.640 INFO:teuthology.orchestra.run.smithi121.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 250 kB/s | 19 kB 00:00 2023-08-25T02:15:20.700 INFO:teuthology.orchestra.run.smithi121.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 716 kB/s | 24 kB 00:00 2023-08-25T02:15:20.727 INFO:teuthology.orchestra.run.smithi121.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 877 kB/s | 29 kB 00:00 2023-08-25T02:15:20.748 INFO:teuthology.orchestra.run.smithi089.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5703.gb 124 kB/s | 15 kB 00:00 2023-08-25T02:15:20.779 INFO:teuthology.orchestra.run.smithi121.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 660 kB/s | 27 kB 00:00 2023-08-25T02:15:20.828 INFO:teuthology.orchestra.run.smithi121.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-08-25T02:15:20.932 INFO:teuthology.orchestra.run.smithi121.stdout:(64/131): ceph-mgr-diskprediction-local-18.0.0- 1.5 MB/s | 7.4 MB 00:04 2023-08-25T02:15:20.942 INFO:teuthology.orchestra.run.smithi121.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2023-08-25T02:15:20.985 INFO:teuthology.orchestra.run.smithi121.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 2.0 MB/s | 85 kB 00:00 2023-08-25T02:15:21.022 INFO:teuthology.orchestra.run.smithi089.stdout:(34/131): ceph-volume-18.0.0-5703.gba6b39e4.el8 948 kB/s | 261 kB 00:00 2023-08-25T02:15:21.043 INFO:teuthology.orchestra.run.smithi121.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 665 kB/s | 61 kB 00:00 2023-08-25T02:15:21.044 INFO:teuthology.orchestra.run.smithi121.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2023-08-25T02:15:21.075 INFO:teuthology.orchestra.run.smithi121.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-08-25T02:15:21.206 INFO:teuthology.orchestra.run.smithi089.stdout:(35/131): cephadm-18.0.0-5703.gba6b39e4.el8.noa 935 kB/s | 171 kB 00:00 2023-08-25T02:15:21.206 INFO:teuthology.orchestra.run.smithi121.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 10 MB/s | 1.7 MB 00:00 2023-08-25T02:15:21.295 INFO:teuthology.orchestra.run.smithi089.stdout:(36/131): cephfs-top-18.0.0-5703.gba6b39e4.el8. 166 kB/s | 18 kB 00:00 2023-08-25T02:15:21.347 INFO:teuthology.orchestra.run.smithi121.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 698 kB/s | 110 kB 00:00 2023-08-25T02:15:21.481 INFO:teuthology.orchestra.run.smithi089.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2023-08-25T02:15:21.547 INFO:teuthology.orchestra.run.smithi089.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-08-25T02:15:21.548 INFO:teuthology.orchestra.run.smithi121.stdout:(72/131): python3-scipy-1.0.0-21.module+el8.5.0 30 MB/s | 14 MB 00:00 2023-08-25T02:15:21.598 INFO:teuthology.orchestra.run.smithi121.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 270 kB/s | 54 kB 00:00 2023-08-25T02:15:21.651 INFO:teuthology.orchestra.run.smithi089.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 932 kB/s | 75 kB 00:00 2023-08-25T02:15:21.664 INFO:teuthology.orchestra.run.smithi121.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2023-08-25T02:15:21.744 INFO:teuthology.orchestra.run.smithi089.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.9 MB/s | 476 kB 00:00 2023-08-25T02:15:21.876 INFO:teuthology.orchestra.run.smithi089.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-08-25T02:15:21.877 INFO:teuthology.orchestra.run.smithi121.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2023-08-25T02:15:21.877 INFO:teuthology.orchestra.run.smithi121.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 245 kB/s | 39 kB 00:00 2023-08-25T02:15:21.877 INFO:teuthology.orchestra.run.smithi121.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2023-08-25T02:15:21.962 INFO:teuthology.orchestra.run.smithi121.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 5.2 MB/s | 538 kB 00:00 2023-08-25T02:15:22.252 INFO:teuthology.orchestra.run.smithi121.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-08-25T02:15:22.258 INFO:teuthology.orchestra.run.smithi121.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-08-25T02:15:22.259 INFO:teuthology.orchestra.run.smithi121.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-08-25T02:15:22.281 INFO:teuthology.orchestra.run.smithi089.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 570 kB/s | 185 kB 00:00 2023-08-25T02:15:22.282 INFO:teuthology.orchestra.run.smithi089.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 661 kB/s | 33 kB 00:00 2023-08-25T02:15:22.282 INFO:teuthology.orchestra.run.smithi089.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 490 kB/s | 16 kB 00:00 2023-08-25T02:15:22.283 INFO:teuthology.orchestra.run.smithi089.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2023-08-25T02:15:22.283 INFO:teuthology.orchestra.run.smithi089.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 7.5 MB/s | 384 kB 00:00 2023-08-25T02:15:22.283 INFO:teuthology.orchestra.run.smithi121.stdout:(82/131): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-08-25T02:15:22.283 INFO:teuthology.orchestra.run.smithi121.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 187 kB/s | 27 kB 00:00 2023-08-25T02:15:22.390 INFO:teuthology.orchestra.run.smithi089.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2023-08-25T02:15:22.471 INFO:teuthology.orchestra.run.smithi089.stdout:(48/131): ceph-mgr-diskprediction-local-18.0.0- 1.5 MB/s | 7.4 MB 00:05 2023-08-25T02:15:22.478 INFO:teuthology.orchestra.run.smithi121.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 191 kB/s | 44 kB 00:00 2023-08-25T02:15:22.497 INFO:teuthology.orchestra.run.smithi089.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-08-25T02:15:22.514 INFO:teuthology.orchestra.run.smithi121.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.1 MB/s | 320 kB 00:00 2023-08-25T02:15:22.594 INFO:teuthology.orchestra.run.smithi089.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2023-08-25T02:15:22.594 INFO:teuthology.orchestra.run.smithi089.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 979 kB/s | 49 kB 00:00 2023-08-25T02:15:22.633 INFO:teuthology.orchestra.run.smithi089.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-08-25T02:15:22.699 INFO:teuthology.orchestra.run.smithi089.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-08-25T02:15:22.699 INFO:teuthology.orchestra.run.smithi121.stdout:(86/131): python3-numpy-1.14.3-10.el8.x86_64.rp 22 MB/s | 3.7 MB 00:00 2023-08-25T02:15:22.751 INFO:teuthology.orchestra.run.smithi121.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.6 MB/s | 457 kB 00:00 2023-08-25T02:15:22.774 INFO:teuthology.orchestra.run.smithi089.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 4.6 MB/s | 1.2 MB 00:00 2023-08-25T02:15:22.774 INFO:teuthology.orchestra.run.smithi089.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-08-25T02:15:22.779 INFO:teuthology.orchestra.run.smithi121.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.9 MB/s | 225 kB 00:00 2023-08-25T02:15:22.797 INFO:teuthology.orchestra.run.smithi089.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-08-25T02:15:22.829 INFO:teuthology.orchestra.run.smithi089.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 576 kB/s | 34 kB 00:00 2023-08-25T02:15:22.845 INFO:teuthology.orchestra.run.smithi089.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-08-25T02:15:22.845 INFO:teuthology.orchestra.run.smithi121.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 426 kB/s | 64 kB 00:00 2023-08-25T02:15:22.876 INFO:teuthology.orchestra.run.smithi089.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-08-25T02:15:22.903 INFO:teuthology.orchestra.run.smithi089.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 370 kB/s | 19 kB 00:00 2023-08-25T02:15:22.936 INFO:teuthology.orchestra.run.smithi089.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2023-08-25T02:15:22.937 INFO:teuthology.orchestra.run.smithi121.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 797 kB/s | 126 kB 00:00 2023-08-25T02:15:23.041 INFO:teuthology.orchestra.run.smithi121.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 184 kB/s | 20 kB 00:00 2023-08-25T02:15:23.041 INFO:teuthology.orchestra.run.smithi089.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 499 kB/s | 29 kB 00:00 2023-08-25T02:15:23.042 INFO:teuthology.orchestra.run.smithi089.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2023-08-25T02:15:23.068 INFO:teuthology.orchestra.run.smithi121.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 826 kB/s | 103 kB 00:00 2023-08-25T02:15:23.068 INFO:teuthology.orchestra.run.smithi089.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2023-08-25T02:15:23.087 INFO:teuthology.orchestra.run.smithi089.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-08-25T02:15:23.133 INFO:teuthology.orchestra.run.smithi121.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 4.0 MB/s | 709 kB 00:00 2023-08-25T02:15:23.134 INFO:teuthology.orchestra.run.smithi089.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 916 kB/s | 61 kB 00:00 2023-08-25T02:15:23.134 INFO:teuthology.orchestra.run.smithi089.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 2.0 MB/s | 85 kB 00:00 2023-08-25T02:15:23.173 INFO:teuthology.orchestra.run.smithi089.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2023-08-25T02:15:23.221 INFO:teuthology.orchestra.run.smithi089.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-08-25T02:15:23.228 INFO:teuthology.orchestra.run.smithi121.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2023-08-25T02:15:23.292 INFO:teuthology.orchestra.run.smithi121.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 20 MB/s | 4.9 MB 00:00 2023-08-25T02:15:23.307 INFO:teuthology.orchestra.run.smithi089.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 9.9 MB/s | 1.7 MB 00:00 2023-08-25T02:15:23.394 INFO:teuthology.orchestra.run.smithi089.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 280 kB/s | 47 kB 00:00 2023-08-25T02:15:23.447 INFO:teuthology.orchestra.run.smithi121.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-08-25T02:15:23.465 INFO:teuthology.orchestra.run.smithi089.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 427 kB/s | 43 kB 00:00 2023-08-25T02:15:23.508 INFO:teuthology.orchestra.run.smithi089.stdout:(73/131): libselinux-devel-2.9-5.el8.x86_64.rpm 997 kB/s | 200 kB 00:00 2023-08-25T02:15:23.539 INFO:teuthology.orchestra.run.smithi121.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-08-25T02:15:23.566 INFO:teuthology.orchestra.run.smithi121.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2023-08-25T02:15:23.645 INFO:teuthology.orchestra.run.smithi089.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2023-08-25T02:15:23.648 INFO:teuthology.orchestra.run.smithi089.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 360 kB/s | 48 kB 00:00 2023-08-25T02:15:23.656 INFO:teuthology.orchestra.run.smithi121.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 8.6 MB/s | 808 kB 00:00 2023-08-25T02:15:23.715 INFO:teuthology.orchestra.run.smithi089.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-08-25T02:15:23.733 INFO:teuthology.orchestra.run.smithi089.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 465 kB/s | 43 kB 00:00 2023-08-25T02:15:23.790 INFO:teuthology.orchestra.run.smithi121.stdout:(100/131): postgresql-server-10.21-2.module+el8 18 MB/s | 5.1 MB 00:00 2023-08-25T02:15:23.850 INFO:teuthology.orchestra.run.smithi089.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 622 kB/s | 109 kB 00:00 2023-08-25T02:15:23.850 INFO:teuthology.orchestra.run.smithi121.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-08-25T02:15:23.885 INFO:teuthology.orchestra.run.smithi089.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-08-25T02:15:23.901 INFO:teuthology.orchestra.run.smithi121.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 432 kB/s | 47 kB 00:00 2023-08-25T02:15:24.002 INFO:teuthology.orchestra.run.smithi089.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-08-25T02:15:24.035 INFO:teuthology.orchestra.run.smithi089.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 118 kB/s | 19 kB 00:00 2023-08-25T02:15:24.057 INFO:teuthology.orchestra.run.smithi121.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 930 kB/s | 200 kB 00:00 2023-08-25T02:15:24.082 INFO:teuthology.orchestra.run.smithi121.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2023-08-25T02:15:24.109 INFO:teuthology.orchestra.run.smithi089.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.4 MB/s | 155 kB 00:00 2023-08-25T02:15:24.171 INFO:teuthology.orchestra.run.smithi089.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 4.7 MB/s | 559 kB 00:00 2023-08-25T02:15:24.238 INFO:teuthology.orchestra.run.smithi121.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2023-08-25T02:15:24.279 INFO:teuthology.orchestra.run.smithi089.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 153 kB/s | 18 kB 00:00 2023-08-25T02:15:24.315 INFO:teuthology.orchestra.run.smithi089.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-08-25T02:15:24.316 INFO:teuthology.orchestra.run.smithi121.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 288 kB/s | 48 kB 00:00 2023-08-25T02:15:24.316 INFO:teuthology.orchestra.run.smithi121.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-08-25T02:15:24.316 INFO:teuthology.orchestra.run.smithi121.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-08-25T02:15:24.363 INFO:teuthology.orchestra.run.smithi121.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.2 MB/s | 109 kB 00:00 2023-08-25T02:15:24.398 INFO:teuthology.orchestra.run.smithi121.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-08-25T02:15:24.398 INFO:teuthology.orchestra.run.smithi089.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 20 MB/s | 2.3 MB 00:00 2023-08-25T02:15:24.458 INFO:teuthology.orchestra.run.smithi089.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2023-08-25T02:15:24.522 INFO:teuthology.orchestra.run.smithi121.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.6 MB/s | 560 kB 00:00 2023-08-25T02:15:24.532 INFO:teuthology.orchestra.run.smithi089.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 764 kB/s | 83 kB 00:00 2023-08-25T02:15:24.540 INFO:teuthology.orchestra.run.smithi121.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2023-08-25T02:15:24.553 INFO:teuthology.orchestra.run.smithi089.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 432 kB/s | 72 kB 00:00 2023-08-25T02:15:24.644 INFO:teuthology.orchestra.run.smithi121.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.2 MB/s | 155 kB 00:00 2023-08-25T02:15:24.656 INFO:teuthology.orchestra.run.smithi089.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 981 kB/s | 98 kB 00:00 2023-08-25T02:15:24.681 INFO:teuthology.orchestra.run.smithi089.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 398 kB/s | 60 kB 00:00 2023-08-25T02:15:24.682 INFO:teuthology.orchestra.run.smithi121.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 4.1 MB/s | 559 kB 00:00 2023-08-25T02:15:24.804 INFO:teuthology.orchestra.run.smithi121.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 153 kB/s | 18 kB 00:00 2023-08-25T02:15:24.823 INFO:teuthology.orchestra.run.smithi089.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 384 kB/s | 58 kB 00:00 2023-08-25T02:15:24.835 INFO:teuthology.orchestra.run.smithi121.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-08-25T02:15:24.866 INFO:teuthology.orchestra.run.smithi089.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 220 kB/s | 39 kB 00:00 2023-08-25T02:15:24.925 INFO:teuthology.orchestra.run.smithi121.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.4 MB/s | 238 kB 00:00 2023-08-25T02:15:24.963 INFO:teuthology.orchestra.run.smithi089.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.5 MB/s | 171 kB 00:00 2023-08-25T02:15:25.024 INFO:teuthology.orchestra.run.smithi089.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 2.9 MB/s | 644 kB 00:00 2023-08-25T02:15:25.024 INFO:teuthology.orchestra.run.smithi121.stdout:(118/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 12 MB/s | 2.3 MB 00:00 2023-08-25T02:15:25.050 INFO:teuthology.orchestra.run.smithi121.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 577 kB/s | 72 kB 00:00 2023-08-25T02:15:25.113 INFO:teuthology.orchestra.run.smithi089.stdout:(96/131): NetworkManager-initscripts-updown-1.3 916 kB/s | 138 kB 00:00 2023-08-25T02:15:25.262 INFO:teuthology.orchestra.run.smithi121.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 451 kB/s | 83 kB 00:00 2023-08-25T02:15:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 326 kB/s | 60 kB 00:00 2023-08-25T02:15:25.405 INFO:teuthology.orchestra.run.smithi121.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 784 kB/s | 98 kB 00:00 2023-08-25T02:15:25.409 INFO:teuthology.orchestra.run.smithi121.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 460 kB/s | 58 kB 00:00 2023-08-25T02:15:25.477 INFO:teuthology.orchestra.run.smithi121.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 272 kB/s | 39 kB 00:00 2023-08-25T02:15:25.595 INFO:teuthology.orchestra.run.smithi121.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.2 MB/s | 644 kB 00:00 2023-08-25T02:15:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:(97/131): ceph-test-18.0.0-5703.gba6b39e4.el8.x 2.0 MB/s | 48 MB 00:23 2023-08-25T02:15:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 236 kB/s | 110 kB 00:00 2023-08-25T02:15:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 362 kB/s | 54 kB 00:00 2023-08-25T02:15:25.756 INFO:teuthology.orchestra.run.smithi121.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.0 MB/s | 171 kB 00:00 2023-08-25T02:15:25.757 INFO:teuthology.orchestra.run.smithi121.stdout:(127/131): NetworkManager-initscripts-updown-1. 868 kB/s | 138 kB 00:00 2023-08-25T02:15:25.832 INFO:teuthology.orchestra.run.smithi089.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-08-25T02:15:25.929 INFO:teuthology.orchestra.run.smithi089.stdout:(101/131): python3-scipy-1.0.0-21.module+el8.5. 15 MB/s | 14 MB 00:00 2023-08-25T02:15:25.949 INFO:teuthology.orchestra.run.smithi089.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 221 kB/s | 39 kB 00:00 2023-08-25T02:15:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:(103/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-08-25T02:15:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 3.7 MB/s | 688 kB 00:00 2023-08-25T02:15:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2023-08-25T02:15:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:(106/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.4 MB/s | 538 kB 00:00 2023-08-25T02:15:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-08-25T02:15:26.321 INFO:teuthology.orchestra.run.smithi089.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-08-25T02:15:26.386 INFO:teuthology.orchestra.run.smithi089.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 187 kB/s | 27 kB 00:00 2023-08-25T02:15:26.436 INFO:teuthology.orchestra.run.smithi089.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-08-25T02:15:26.470 INFO:teuthology.orchestra.run.smithi089.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2023-08-25T02:15:26.526 INFO:teuthology.orchestra.run.smithi089.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 250 kB/s | 44 kB 00:00 2023-08-25T02:15:26.648 INFO:teuthology.orchestra.run.smithi089.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 18 MB/s | 3.7 MB 00:00 2023-08-25T02:15:26.684 INFO:teuthology.orchestra.run.smithi089.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-08-25T02:15:26.707 INFO:teuthology.orchestra.run.smithi089.stdout:(115/131): python3-werkzeug-0.12.2-4.el8.noarch 1.8 MB/s | 457 kB 00:00 2023-08-25T02:15:26.814 INFO:teuthology.orchestra.run.smithi089.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 946 kB/s | 126 kB 00:00 2023-08-25T02:15:26.854 INFO:teuthology.orchestra.run.smithi089.stdout:(117/131): uuid-1.6.2-43.el8.x86_64.rpm 318 kB/s | 64 kB 00:00 2023-08-25T02:15:26.889 INFO:teuthology.orchestra.run.smithi089.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 114 kB/s | 20 kB 00:00 2023-08-25T02:15:26.918 INFO:teuthology.orchestra.run.smithi089.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 950 kB/s | 103 kB 00:00 2023-08-25T02:15:27.013 INFO:teuthology.orchestra.run.smithi089.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2023-08-25T02:15:27.114 INFO:teuthology.orchestra.run.smithi089.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 21 MB/s | 4.9 MB 00:00 2023-08-25T02:15:27.149 INFO:teuthology.orchestra.run.smithi089.stdout:(122/131): libpq-13.5-1.el8.x86_64.rpm 876 kB/s | 198 kB 00:00 2023-08-25T02:15:27.224 INFO:teuthology.orchestra.run.smithi089.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-08-25T02:15:27.282 INFO:teuthology.orchestra.run.smithi089.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-08-25T02:15:27.385 INFO:teuthology.orchestra.run.smithi089.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-08-25T02:15:27.491 INFO:teuthology.orchestra.run.smithi089.stdout:(126/131): postgresql-contrib-10.21-2.module+el 3.8 MB/s | 808 kB 00:00 2023-08-25T02:15:27.605 INFO:teuthology.orchestra.run.smithi089.stdout:(127/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-08-25T02:15:27.610 INFO:teuthology.orchestra.run.smithi089.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.4 MB/s | 1.5 MB 00:00 2023-08-25T02:15:27.673 INFO:teuthology.orchestra.run.smithi121.stdout:(128/131): librados2-18.0.0-5703.gba6b39e4.el8. 1.8 MB/s | 3.7 MB 00:02 2023-08-25T02:15:27.802 INFO:teuthology.orchestra.run.smithi089.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.1 MB/s | 798 kB 00:00 2023-08-25T02:15:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:(129/131): ceph-test-18.0.0-5703.gba6b39e4.el8. 2.1 MB/s | 48 MB 00:22 2023-08-25T02:15:28.224 INFO:teuthology.orchestra.run.smithi121.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 1.6 MB/s | 798 kB 00:00 2023-08-25T02:15:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:(131/131): librbd1-18.0.0-5703.gba6b39e4.el8.x8 1.2 MB/s | 3.6 MB 00:03 2023-08-25T02:15:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:15:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Total 7.0 MB/s | 206 MB 00:29 2023-08-25T02:15:28.824 INFO:teuthology.orchestra.run.smithi089.stdout:(130/131): librbd1-18.0.0-5703.gba6b39e4.el8.x8 3.0 MB/s | 3.6 MB 00:01 2023-08-25T02:15:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-08-25T02:15:29.576 INFO:teuthology.orchestra.run.smithi122.stdout:(131/131): ceph-test-18.0.0-5703.gba6b39e4.el8. 1.5 MB/s | 48 MB 00:31 2023-08-25T02:15:29.659 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:15:29.659 INFO:teuthology.orchestra.run.smithi122.stdout:Total 6.0 MB/s | 206 MB 00:34 2023-08-25T02:15:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-08-25T02:15:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-08-25T02:15:29.934 INFO:teuthology.orchestra.run.smithi089.stdout:(131/131): librados2-18.0.0-5703.gba6b39e4.el8. 1.5 MB/s | 3.7 MB 00:02 2023-08-25T02:15:29.964 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:15:29.965 INFO:teuthology.orchestra.run.smithi089.stdout:Total 6.2 MB/s | 206 MB 00:32 2023-08-25T02:15:30.403 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2023-08-25T02:15:30.523 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2023-08-25T02:15:30.523 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2023-08-25T02:15:30.764 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-08-25T02:15:30.809 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-08-25T02:15:30.810 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-08-25T02:15:30.859 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-08-25T02:15:30.860 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-08-25T02:15:31.649 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2023-08-25T02:15:31.677 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2023-08-25T02:15:31.987 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-08-25T02:15:31.987 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-08-25T02:15:35.957 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-08-25T02:15:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-08-25T02:15:36.333 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-08-25T02:15:36.534 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-08-25T02:15:36.717 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2023-08-25T02:15:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-08-25T02:15:36.864 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-08-25T02:15:36.918 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-08-25T02:15:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-08-25T02:15:37.094 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-08-25T02:15:37.110 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-08-25T02:15:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-08-25T02:15:37.333 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-08-25T02:15:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-08-25T02:15:37.423 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-08-25T02:15:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-08-25T02:15:37.545 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-08-25T02:15:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-08-25T02:15:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-08-25T02:15:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-08-25T02:15:38.088 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-08-25T02:15:38.163 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-08-25T02:15:38.365 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 5/134 2023-08-25T02:15:38.398 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 5/134 2023-08-25T02:15:38.495 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:38.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:15:38.762 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 5/134 2023-08-25T02:15:38.785 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:15:38.815 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 5/134 2023-08-25T02:15:38.990 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:38.991 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 9/134 2023-08-25T02:15:38.991 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:39.071 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:39.132 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:15:39.236 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-08-25T02:15:39.309 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:39.325 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:39.401 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:15:39.451 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-08-25T02:15:39.560 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:15:39.579 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:15:39.712 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-08-25T02:15:39.751 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-08-25T02:15:39.804 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-08-25T02:15:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-08-25T02:15:40.025 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-08-25T02:15:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-08-25T02:15:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-08-25T02:15:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:40.260 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-08-25T02:15:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:40.580 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-08-25T02:15:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:40.693 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-08-25T02:15:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-08-25T02:15:40.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-08-25T02:15:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-08-25T02:15:40.937 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-08-25T02:15:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-08-25T02:15:41.070 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:41.075 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:41.182 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 14/134 2023-08-25T02:15:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:41.361 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:41.473 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 15/134 2023-08-25T02:15:41.599 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-08-25T02:15:41.658 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-08-25T02:15:41.713 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-08-25T02:15:41.757 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-08-25T02:15:41.832 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-08-25T02:15:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-08-25T02:15:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-08-25T02:15:42.037 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-08-25T02:15:42.115 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-08-25T02:15:42.176 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-08-25T02:15:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-08-25T02:15:42.313 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-08-25T02:15:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-08-25T02:15:42.483 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-08-25T02:15:42.610 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-08-25T02:15:42.633 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-08-25T02:15:42.950 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-08-25T02:15:43.097 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-08-25T02:15:43.221 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-08-25T02:15:43.347 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-08-25T02:15:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-08-25T02:15:43.692 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-08-25T02:15:43.792 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-08-25T02:15:43.856 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-08-25T02:15:44.018 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-08-25T02:15:44.052 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-08-25T02:15:44.165 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-08-25T02:15:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-08-25T02:15:44.520 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-08-25T02:15:44.542 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-08-25T02:15:44.713 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-08-25T02:15:44.776 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-08-25T02:15:44.896 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-08-25T02:15:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-08-25T02:15:44.992 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-08-25T02:15:45.140 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-08-25T02:15:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 26/134 2023-08-25T02:15:45.221 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-08-25T02:15:45.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 27/134 2023-08-25T02:15:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 26/134 2023-08-25T02:15:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 27/134 2023-08-25T02:15:45.597 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-08-25T02:15:45.747 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 28/134 2023-08-25T02:15:45.876 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-08-25T02:15:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-08-25T02:15:46.063 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 26/134 2023-08-25T02:15:46.183 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 27/134 2023-08-25T02:15:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 28/134 2023-08-25T02:15:46.431 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 28/134 2023-08-25T02:15:46.520 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-08-25T02:15:46.648 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-08-25T02:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-08-25T02:15:47.030 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-08-25T02:15:47.261 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-08-25T02:15:47.675 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-08-25T02:15:48.397 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-08-25T02:15:48.668 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-08-25T02:15:48.984 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-08-25T02:15:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-08-25T02:15:49.264 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-08-25T02:15:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-08-25T02:15:49.454 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-08-25T02:15:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-08-25T02:15:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-08-25T02:15:49.689 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-08-25T02:15:49.748 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-08-25T02:15:49.770 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-08-25T02:15:49.873 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-08-25T02:15:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 36/134 2023-08-25T02:15:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 36/134 2023-08-25T02:15:50.100 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-08-25T02:15:50.135 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 37/134 2023-08-25T02:15:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 37/134 2023-08-25T02:15:50.324 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-08-25T02:15:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-08-25T02:15:50.362 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-08-25T02:15:50.512 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-08-25T02:15:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-08-25T02:15:50.601 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-08-25T02:15:50.680 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-08-25T02:15:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-08-25T02:15:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-08-25T02:15:50.824 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-08-25T02:15:50.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-08-25T02:15:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-08-25T02:15:51.045 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 36/134 2023-08-25T02:15:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-08-25T02:15:51.191 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-08-25T02:15:51.192 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 37/134 2023-08-25T02:15:51.287 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-08-25T02:15:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-08-25T02:15:51.363 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-08-25T02:15:51.392 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-08-25T02:15:51.515 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-08-25T02:15:51.615 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-08-25T02:15:51.689 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-08-25T02:15:51.742 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-08-25T02:15:51.791 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-08-25T02:15:51.796 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-08-25T02:15:51.942 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-08-25T02:15:51.991 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-08-25T02:15:52.029 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-08-25T02:15:52.113 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-08-25T02:15:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-08-25T02:15:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-08-25T02:15:52.270 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-08-25T02:15:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-08-25T02:15:52.441 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-08-25T02:15:52.456 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-08-25T02:15:52.500 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-08-25T02:15:52.668 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-08-25T02:15:52.685 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-08-25T02:15:52.727 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-08-25T02:15:52.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-08-25T02:15:52.876 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-08-25T02:15:53.047 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-08-25T02:15:53.072 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-08-25T02:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-08-25T02:15:53.248 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-08-25T02:15:53.263 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-08-25T02:15:53.440 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-08-25T02:15:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-08-25T02:15:53.537 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-08-25T02:15:53.626 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-08-25T02:15:53.677 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-08-25T02:15:53.757 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-08-25T02:15:53.873 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-08-25T02:15:53.905 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-08-25T02:15:53.973 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-08-25T02:15:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-08-25T02:15:54.145 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-08-25T02:15:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-08-25T02:15:54.336 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-08-25T02:15:54.555 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-08-25T02:15:54.568 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-08-25T02:15:54.620 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-08-25T02:15:54.824 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-08-25T02:15:54.883 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-08-25T02:15:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-08-25T02:15:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-08-25T02:15:55.104 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-08-25T02:15:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-08-25T02:15:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-08-25T02:15:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-08-25T02:15:55.357 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-08-25T02:15:55.461 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-08-25T02:15:55.461 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-08-25T02:15:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-08-25T02:15:55.680 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-08-25T02:15:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-08-25T02:15:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-08-25T02:15:55.792 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-08-25T02:15:56.008 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-08-25T02:15:56.069 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-08-25T02:15:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-08-25T02:15:56.248 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-08-25T02:15:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-08-25T02:15:56.283 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-08-25T02:15:56.430 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-08-25T02:15:56.460 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-08-25T02:15:56.499 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-08-25T02:15:56.520 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-08-25T02:15:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-08-25T02:15:56.702 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-08-25T02:15:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-08-25T02:15:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-08-25T02:15:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-08-25T02:15:56.906 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-08-25T02:15:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-08-25T02:15:57.101 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-08-25T02:15:57.101 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-08-25T02:15:57.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-08-25T02:15:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-08-25T02:15:57.369 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-08-25T02:15:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-08-25T02:15:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-08-25T02:15:57.617 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-08-25T02:15:57.643 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-08-25T02:15:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-08-25T02:15:57.833 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-08-25T02:15:57.952 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-08-25T02:15:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-08-25T02:15:58.068 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-08-25T02:15:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-08-25T02:15:58.220 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-08-25T02:15:58.256 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-08-25T02:15:58.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-08-25T02:15:58.372 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-08-25T02:15:58.422 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-08-25T02:15:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-08-25T02:15:58.567 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-08-25T02:15:58.584 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-08-25T02:15:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-08-25T02:15:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-08-25T02:15:58.872 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-08-25T02:15:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-08-25T02:15:58.995 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-08-25T02:15:59.119 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-08-25T02:15:59.120 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-08-25T02:15:59.215 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-08-25T02:15:59.276 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-08-25T02:15:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-08-25T02:15:59.454 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-08-25T02:15:59.457 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-08-25T02:15:59.628 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-08-25T02:15:59.656 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-08-25T02:15:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-08-25T02:15:59.792 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-08-25T02:15:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-08-25T02:15:59.901 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-08-25T02:16:00.056 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-08-25T02:16:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-08-25T02:16:00.073 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-08-25T02:16:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-08-25T02:16:00.300 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-08-25T02:16:00.433 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-08-25T02:16:00.435 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-08-25T02:16:00.678 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-08-25T02:16:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-08-25T02:16:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-08-25T02:16:01.156 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-08-25T02:16:01.187 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-08-25T02:16:01.422 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-08-25T02:16:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-08-25T02:16:01.671 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-08-25T02:16:01.688 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-08-25T02:16:01.796 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-08-25T02:16:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-08-25T02:16:01.991 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-08-25T02:16:02.101 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-08-25T02:16:02.137 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-08-25T02:16:02.258 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-08-25T02:16:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-08-25T02:16:02.467 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-08-25T02:16:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-08-25T02:16:03.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-08-25T02:16:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-08-25T02:16:03.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-08-25T02:16:03.242 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-08-25T02:16:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:03.483 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-08-25T02:16:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 93/134 2023-08-25T02:16:03.754 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-08-25T02:16:04.044 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-08-25T02:16:04.106 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-08-25T02:16:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:04.414 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-08-25T02:16:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 93/134 2023-08-25T02:16:04.635 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-08-25T02:16:05.406 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-08-25T02:16:05.621 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:05.685 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 92/134 2023-08-25T02:16:05.888 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 93/134 2023-08-25T02:16:07.296 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:07.544 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:08.429 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:09.232 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:09.406 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:09.449 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:09.605 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:09.870 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:09.912 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:10.444 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 94/134 2023-08-25T02:16:10.771 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:10.881 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 95/134 2023-08-25T02:16:11.025 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:28.272 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2023-08-25T02:16:28.272 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2023-08-25T02:16:28.272 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2023-08-25T02:16:28.272 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /run 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2023-08-25T02:16:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:16:28.625 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-08-25T02:16:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-08-25T02:16:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-08-25T02:16:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2023-08-25T02:16:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-08-25T02:16:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-08-25T02:16:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-08-25T02:16:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-08-25T02:16:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-08-25T02:16:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:16:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:29.804 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 96/134 2023-08-25T02:16:29.805 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2023-08-25T02:16:29.805 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2023-08-25T02:16:29.805 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2023-08-25T02:16:29.805 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /run 2023-08-25T02:16:29.805 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2023-08-25T02:16:29.806 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2023-08-25T02:16:29.806 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2023-08-25T02:16:29.806 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2023-08-25T02:16:29.806 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2023-08-25T02:16:29.806 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:16:30.184 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:30.200 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:30.362 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:30.697 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:30.739 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:30.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:31.108 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:31.192 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:31.291 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:31.363 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:31.740 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:31.756 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 97/134 2023-08-25T02:16:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:31.977 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:32.192 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 98/134 2023-08-25T02:16:32.368 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:32.820 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 99/134 2023-08-25T02:16:32.996 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:33.011 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 100/134 2023-08-25T02:16:33.040 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:33.206 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:33.370 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 102/134 2023-08-25T02:16:33.570 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 103/134 2023-08-25T02:16:33.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:33.753 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:33.974 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:34.017 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 102/134 2023-08-25T02:16:34.223 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 103/134 2023-08-25T02:16:34.261 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:34.624 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:34.734 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 101/134 2023-08-25T02:16:34.952 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:35.053 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 102/134 2023-08-25T02:16:35.216 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:35.260 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 103/134 2023-08-25T02:16:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:35.481 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:35.696 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 104/134 2023-08-25T02:16:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 107/134 2023-08-25T02:16:35.901 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:36.095 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:36.533 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 107/134 2023-08-25T02:16:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:36.785 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:36.946 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:36.960 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 105/134 2023-08-25T02:16:37.199 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:37.291 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 106/134 2023-08-25T02:16:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 110/134 2023-08-25T02:16:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 107/134 2023-08-25T02:16:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 108/134 2023-08-25T02:16:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:38.061 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:38.070 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 109/134 2023-08-25T02:16:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 110/134 2023-08-25T02:16:38.400 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:38.409 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:38.450 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:38.562 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:38.628 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:38.676 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:38.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:39.198 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:39.389 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 110/134 2023-08-25T02:16:39.397 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:39.635 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 111/134 2023-08-25T02:16:39.858 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:39.900 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 112/134 2023-08-25T02:16:40.050 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 113/134 2023-08-25T02:16:40.585 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:43.115 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:43.951 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:44.000 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 115/134 2023-08-25T02:16:44.278 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 115/134 2023-08-25T02:16:44.335 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-08-25T02:16:44.462 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-08-25T02:16:44.505 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 117/134 2023-08-25T02:16:44.650 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-08-25T02:16:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-08-25T02:16:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-08-25T02:16:45.074 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-08-25T02:16:45.135 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 114/134 2023-08-25T02:16:45.282 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-08-25T02:16:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 121/134 2023-08-25T02:16:45.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-08-25T02:16:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-08-25T02:16:45.867 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-08-25T02:16:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 119/134 2023-08-25T02:16:46.063 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 115/134 2023-08-25T02:16:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-08-25T02:16:46.367 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 121/134 2023-08-25T02:16:46.421 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-08-25T02:16:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-08-25T02:16:46.620 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 117/134 2023-08-25T02:16:46.903 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-08-25T02:16:47.057 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-08-25T02:16:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-08-25T02:16:47.494 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-08-25T02:16:47.661 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 121/134 2023-08-25T02:16:47.796 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:47.847 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-08-25T02:16:48.026 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:48.043 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:48.194 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 124/134 2023-08-25T02:16:48.526 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 125/134 2023-08-25T02:16:48.801 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 126/134 2023-08-25T02:16:48.802 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:49.125 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-08-25T02:16:49.346 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-08-25T02:16:49.373 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 124/134 2023-08-25T02:16:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-08-25T02:16:49.577 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-08-25T02:16:49.589 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-08-25T02:16:49.608 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 125/134 2023-08-25T02:16:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-08-25T02:16:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-08-25T02:16:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-08-25T02:16:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 126/134 2023-08-25T02:16:50.121 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:50.346 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-08-25T02:16:50.386 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:50.423 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-08-25T02:16:50.508 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-08-25T02:16:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-08-25T02:16:50.677 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 124/134 2023-08-25T02:16:50.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-08-25T02:16:50.907 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 125/134 2023-08-25T02:16:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-08-25T02:16:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-08-25T02:16:51.072 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:51.073 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 1/134 2023-08-25T02:16:51.073 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2/134 2023-08-25T02:16:51.080 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 3/134 2023-08-25T02:16:51.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 4/134 2023-08-25T02:16:51.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 5/134 2023-08-25T02:16:51.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:16:51.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:16:51.081 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:16:51.082 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 9/134 2023-08-25T02:16:51.082 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 10/134 2023-08-25T02:16:51.082 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 11/134 2023-08-25T02:16:51.097 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 12/134 2023-08-25T02:16:51.097 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 13/134 2023-08-25T02:16:51.103 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 14/134 2023-08-25T02:16:51.106 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 15/134 2023-08-25T02:16:51.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 16/134 2023-08-25T02:16:51.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 17/134 2023-08-25T02:16:51.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 18/134 2023-08-25T02:16:51.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 19/134 2023-08-25T02:16:51.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 20/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 21/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 22/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 23/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 24/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 25/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 26/134 2023-08-25T02:16:51.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 27/134 2023-08-25T02:16:51.139 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 28/134 2023-08-25T02:16:51.139 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 29/134 2023-08-25T02:16:51.139 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 30/134 2023-08-25T02:16:51.139 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 31/134 2023-08-25T02:16:51.139 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 32/134 2023-08-25T02:16:51.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 33/134 2023-08-25T02:16:51.176 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 34/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 35/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 36/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 37/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 38/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-08-25T02:16:51.177 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-08-25T02:16:51.178 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-08-25T02:16:51.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-08-25T02:16:51.180 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-08-25T02:16:51.180 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-08-25T02:16:51.180 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-08-25T02:16:51.217 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-08-25T02:16:51.218 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-08-25T02:16:51.218 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-08-25T02:16:51.221 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-08-25T02:16:51.221 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-08-25T02:16:51.222 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-08-25T02:16:51.222 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-08-25T02:16:51.225 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-08-25T02:16:51.226 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-08-25T02:16:51.226 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-08-25T02:16:51.226 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-08-25T02:16:51.226 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-08-25T02:16:51.226 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-08-25T02:16:51.229 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-08-25T02:16:51.229 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-08-25T02:16:51.229 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-08-25T02:16:51.229 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-08-25T02:16:51.229 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-08-25T02:16:51.230 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-08-25T02:16:51.230 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-08-25T02:16:51.230 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-08-25T02:16:51.230 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-08-25T02:16:51.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-08-25T02:16:51.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-08-25T02:16:51.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-08-25T02:16:51.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-08-25T02:16:51.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-08-25T02:16:51.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-08-25T02:16:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-08-25T02:16:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-08-25T02:16:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-08-25T02:16:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-08-25T02:16:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-08-25T02:16:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-08-25T02:16:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-08-25T02:16:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-08-25T02:16:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-08-25T02:16:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-08-25T02:16:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-08-25T02:16:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-08-25T02:16:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-08-25T02:16:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-08-25T02:16:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-08-25T02:16:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-08-25T02:16:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-08-25T02:16:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 129/134 2023-08-25T02:16:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-08-25T02:16:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 131/134 2023-08-25T02:16:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-08-25T02:16:51.288 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 126/134 2023-08-25T02:16:51.295 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-08-25T02:16:51.312 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-08-25T02:16:51.313 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:51.428 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:51.429 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:51.563 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-08-25T02:16:51.673 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:51.734 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-08-25T02:16:51.833 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-08-25T02:16:51.898 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:52.007 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-08-25T02:16:52.027 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-08-25T02:16:52.207 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-08-25T02:16:52.217 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-08-25T02:16:52.371 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-08-25T02:16:52.375 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 1/134 2023-08-25T02:16:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2/134 2023-08-25T02:16:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 3/134 2023-08-25T02:16:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 4/134 2023-08-25T02:16:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 5/134 2023-08-25T02:16:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:16:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:16:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:16:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 9/134 2023-08-25T02:16:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 10/134 2023-08-25T02:16:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 11/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 12/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 13/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 14/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 15/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 16/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 17/134 2023-08-25T02:16:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 18/134 2023-08-25T02:16:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 19/134 2023-08-25T02:16:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 20/134 2023-08-25T02:16:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 21/134 2023-08-25T02:16:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 22/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 23/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 24/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 25/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 26/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 27/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 28/134 2023-08-25T02:16:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 29/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 30/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 31/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 32/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 33/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 34/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 35/134 2023-08-25T02:16:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 36/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 37/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 38/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-08-25T02:16:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-08-25T02:16:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-08-25T02:16:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-08-25T02:16:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-08-25T02:16:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-08-25T02:16:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-08-25T02:16:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-08-25T02:16:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-08-25T02:16:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-08-25T02:16:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-08-25T02:16:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-08-25T02:16:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-08-25T02:16:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-08-25T02:16:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-08-25T02:16:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-08-25T02:16:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-08-25T02:16:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-08-25T02:16:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-08-25T02:16:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-08-25T02:16:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-08-25T02:16:52.510 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-08-25T02:16:52.510 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-08-25T02:16:52.510 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-08-25T02:16:52.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 129/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 131/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:52.512 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-08-25T02:16:52.606 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:52.606 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:52.846 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-08-25T02:16:53.057 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:53.553 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:53.553 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:09 ago on Fri 25 Aug 2023 02:14:44 AM UTC. 2023-08-25T02:16:53.660 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:53.660 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 1/134 2023-08-25T02:16:53.660 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 3/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 4/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 5/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 6/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 7/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 8/134 2023-08-25T02:16:53.661 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 9/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 10/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 11/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 12/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 13/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_ 14/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 15/134 2023-08-25T02:16:53.662 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 16/134 2023-08-25T02:16:53.663 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_6 17/134 2023-08-25T02:16:53.663 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86 18/134 2023-08-25T02:16:53.663 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 19/134 2023-08-25T02:16:53.664 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 20/134 2023-08-25T02:16:53.665 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8. 21/134 2023-08-25T02:16:53.665 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_6 22/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 23/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 24/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 25/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 26/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 27/134 2023-08-25T02:16:53.666 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 28/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 29/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noa 30/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.n 31/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 32/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 33/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 34/134 2023-08-25T02:16:53.667 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 35/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 36/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 37/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 38/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-08-25T02:16:53.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-08-25T02:16:53.669 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-08-25T02:16:53.670 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-08-25T02:16:53.671 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-08-25T02:16:53.671 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-08-25T02:16:53.718 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-08-25T02:16:53.719 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-08-25T02:16:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-08-25T02:16:53.721 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-08-25T02:16:53.722 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-08-25T02:16:53.723 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-08-25T02:16:53.724 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-08-25T02:16:53.724 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-08-25T02:16:53.746 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-08-25T02:16:53.746 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-08-25T02:16:53.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-08-25T02:16:53.748 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-08-25T02:16:53.749 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-08-25T02:16:53.755 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-08-25T02:16:53.755 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-08-25T02:16:53.755 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-08-25T02:16:53.755 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-08-25T02:16:53.755 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 129/134 2023-08-25T02:16:53.756 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-08-25T02:16:53.760 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 131/134 2023-08-25T02:16:53.760 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-08-25T02:16:53.760 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-08-25T02:16:54.947 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:54.947 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:07 ago on Fri 25 Aug 2023 02:14:47 AM UTC. 2023-08-25T02:16:56.162 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-08-25T02:16:56.163 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:13 ago on Fri 25 Aug 2023 02:14:43 AM UTC. 2023-08-25T02:17:01.154 INFO:teuthology.orchestra.run.smithi089.stdout:Installed products updated. 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.373 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-08-25T02:17:01.374 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-08-25T02:17:01.374 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.1-7.el8.noarch 2023-08-25T02:17:01.374 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.0.4-10.el8.x86_64 2023-08-25T02:17:01.374 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.381 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.382 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.383 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.392 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.1-9.el8.x86_64 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.7-15.el8.x86_64 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-25T02:17:01.399 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-3.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.400 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: libpq-13.5-1.el8.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:01.401 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-08-25T02:17:01.402 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-25.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-25T02:17:01.403 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-25T02:17:01.404 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-08-25T02:17:01.404 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-08-25T02:17:01.410 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-08-25T02:17:01.410 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-08-25T02:17:01.411 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:01.413 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:01.413 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:01.413 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.414 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-25T02:17:01.415 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-25T02:17:01.422 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-25T02:17:01.422 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-25T02:17:01.423 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-25T02:17:01.423 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-25T02:17:01.423 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-25T02:17:01.423 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-25T02:17:01.429 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-25T02:17:01.429 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-25T02:17:01.429 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-25T02:17:01.429 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-25T02:17:01.432 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:01.433 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:01.433 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-25T02:17:01.433 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-25T02:17:01.433 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-2.6-1.el8.noarch 2023-08-25T02:17:01.440 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-25T02:17:01.441 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-25T02:17:01.441 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-25T02:17:01.441 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-25T02:17:01.441 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-25T02:17:01.441 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-25T02:17:01.442 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.442 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.443 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-25T02:17:01.443 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-25T02:17:01.443 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.444 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-25T02:17:01.444 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-25T02:17:01.445 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-25T02:17:01.446 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-25T02:17:01.446 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-25T02:17:01.446 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-25T02:17:01.446 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-25T02:17:01.447 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-25T02:17:01.447 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-25T02:17:01.447 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-25T02:17:01.448 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-25T02:17:01.448 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-25T02:17:01.449 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-43.el8.x86_64 2023-08-25T02:17:01.450 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-08-25T02:17:01.450 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-25T02:17:01.450 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-08-25T02:17:01.450 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:17:01.467 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-08-25T02:17:01.573 DEBUG:teuthology.parallel:result is None 2023-08-25T02:17:03.609 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-08-25T02:17:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:17:03.822 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-08-25T02:17:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-08-25T02:17:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.1-7.el8.noarch 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.0.4-10.el8.x86_64 2023-08-25T02:17:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-08-25T02:17:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.1-9.el8.x86_64 2023-08-25T02:17:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.7-15.el8.x86_64 2023-08-25T02:17:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-25T02:17:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-3.el8.x86_64 2023-08-25T02:17:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-08-25T02:17:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-08-25T02:17:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-08-25T02:17:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.888 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.904 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: libpq-13.5-1.el8.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-25.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-25T02:17:03.906 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-08-25T02:17:03.907 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-08-25T02:17:03.907 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-25T02:17:03.907 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-25T02:17:03.907 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-25T02:17:03.907 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:03.920 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:03.920 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-25T02:17:03.920 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-25T02:17:03.926 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-25T02:17:03.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-25T02:17:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-25T02:17:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-25T02:17:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-25T02:17:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-25T02:17:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-25T02:17:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-25T02:17:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-25T02:17:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-25T02:17:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-25T02:17:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-25T02:17:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-25T02:17:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-25T02:17:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-25T02:17:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-25T02:17:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-25T02:17:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-25T02:17:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-25T02:17:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-25T02:17:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-25T02:17:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-25T02:17:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-25T02:17:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-25T02:17:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-25T02:17:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-43.el8.x86_64 2023-08-25T02:17:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-08-25T02:17:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-25T02:17:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-08-25T02:17:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-25T02:17:03.991 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-08-25T02:17:04.599 DEBUG:teuthology.parallel:result is None 2023-08-25T02:17:05.306 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2023-08-25T02:17:05.521 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:17:05.522 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2023-08-25T02:17:05.522 INFO:teuthology.orchestra.run.smithi122.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-08-25T02:17:05.522 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.522 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.523 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2023-08-25T02:17:05.523 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-08-25T02:17:05.523 INFO:teuthology.orchestra.run.smithi122.stdout: automake-1.16.1-7.el8.noarch 2023-08-25T02:17:05.523 INFO:teuthology.orchestra.run.smithi122.stdout: bison-3.0.4-10.el8.x86_64 2023-08-25T02:17:05.523 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.550 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.550 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.551 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.551 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.551 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.551 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.555 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.556 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.557 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.557 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.557 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.557 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.558 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top-2:18.0.0-5703.gba6b39e4.el8.noarch 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-08-25T02:17:05.559 INFO:teuthology.orchestra.run.smithi122.stdout: flex-2.6.1-9.el8.x86_64 2023-08-25T02:17:05.560 INFO:teuthology.orchestra.run.smithi122.stdout: fuse-2.9.7-15.el8.x86_64 2023-08-25T02:17:05.561 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-25T02:17:05.561 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-3.el8.x86_64 2023-08-25T02:17:05.561 INFO:teuthology.orchestra.run.smithi122.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-08-25T02:17:05.561 INFO:teuthology.orchestra.run.smithi122.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-08-25T02:17:05.561 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-08-25T02:17:05.562 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.562 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.562 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.562 INFO:teuthology.orchestra.run.smithi122.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-08-25T02:17:05.562 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: libpq-13.5-1.el8.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.563 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.564 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-25T02:17:05.564 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.566 INFO:teuthology.orchestra.run.smithi122.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-08-25T02:17:05.567 INFO:teuthology.orchestra.run.smithi122.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-08-25T02:17:05.567 INFO:teuthology.orchestra.run.smithi122.stdout: libtool-2.4.6-25.el8.x86_64 2023-08-25T02:17:05.567 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-25T02:17:05.567 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-08-25T02:17:05.568 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-25T02:17:05.569 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-25T02:17:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-25T02:17:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-25T02:17:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-25T02:17:05.570 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.571 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.571 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.571 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-25T02:17:05.571 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-25T02:17:05.572 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-25T02:17:05.572 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-25T02:17:05.572 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-25T02:17:05.613 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-25T02:17:05.614 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-25T02:17:05.615 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:05.616 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-25T02:17:05.616 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-25T02:17:05.636 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-25T02:17:05.637 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-2.6-1.el8.noarch 2023-08-25T02:17:05.637 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-25T02:17:05.637 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-25T02:17:05.637 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-25T02:17:05.637 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-25T02:17:05.638 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-25T02:17:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-25T02:17:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-25T02:17:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-25T02:17:05.673 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-25T02:17:05.674 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-25T02:17:05.675 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-25T02:17:05.675 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-25T02:17:05.675 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.687 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.687 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:18.0.0-5703.gba6b39e4.el8.x86_64 2023-08-25T02:17:05.687 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-25T02:17:05.688 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-25T02:17:05.688 INFO:teuthology.orchestra.run.smithi122.stdout: uuid-1.6.2-43.el8.x86_64 2023-08-25T02:17:05.688 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-08-25T02:17:05.690 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-25T02:17:05.691 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-08-25T02:17:05.691 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:17:05.691 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-08-25T02:17:05.706 DEBUG:teuthology.parallel:result is None 2023-08-25T02:17:05.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:06.015 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-25T02:17:06.111 INFO:teuthology.orchestra.run.smithi089.stdout:18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.111 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.113 INFO:teuthology.task.install:The correct ceph version 18.0.0-5703.gba6b39e4 is installed. 2023-08-25T02:17:06.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:06.533 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-25T02:17:06.642 INFO:teuthology.orchestra.run.smithi121.stdout:18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.649 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.649 INFO:teuthology.task.install:The correct ceph version 18.0.0-5703.gba6b39e4 is installed. 2023-08-25T02:17:06.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:06.945 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-25T02:17:06.981 INFO:teuthology.orchestra.run.smithi122.stdout:18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.981 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5703.gba6b39e4.el8 2023-08-25T02:17:06.982 INFO:teuthology.task.install:The correct ceph version 18.0.0-5703.gba6b39e4 is installed. 2023-08-25T02:17:06.982 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-08-25T02:17:06.984 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:17:06.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-25T02:17:07.043 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:17:07.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-25T02:17:07.190 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:17:07.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-25T02:17:07.329 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-08-25T02:17:07.346 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:17:07.346 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2023-08-25T02:17:07.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-25T02:17:07.535 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:17:07.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-08-25T02:17:07.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-25T02:17:07.821 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:17:07.822 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2023-08-25T02:17:07.907 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-25T02:17:07.982 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-08-25T02:17:07.982 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:17:07.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-25T02:17:08.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-25T02:17:08.117 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:17:08.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-25T02:17:08.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-25T02:17:08.267 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:17:08.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-25T02:17:08.341 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-25T02:17:08.426 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-08-25T02:17:08.426 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:17:08.477 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2023-08-25T02:17:08.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-25T02:17:08.629 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:17:08.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2023-08-25T02:17:08.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-25T02:17:08.743 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:17:08.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2023-08-25T02:17:08.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-25T02:17:08.892 INFO:teuthology.run_tasks:Running task cephadm... 2023-08-25T02:17:09.472 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'} 2023-08-25T02:17:09.473 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:09.570 INFO:tasks.cephadm:Cluster fsid is 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:17:09.595 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-08-25T02:17:09.596 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.89', 'mon.b': '172.21.15.121', 'mon.c': '172.21.15.122'} 2023-08-25T02:17:09.596 INFO:tasks.cephadm:First mon is mon.a on smithi089 2023-08-25T02:17:09.597 INFO:tasks.cephadm:First mgr is x 2023-08-25T02:17:09.597 INFO:tasks.cephadm:Normalizing hostnames... 2023-08-25T02:17:09.597 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2023-08-25T02:17:09.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2023-08-25T02:17:09.706 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2023-08-25T02:17:09.753 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-08-25T02:17:09.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:10.059 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72276/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5703-gba6b39e4', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5703.gba6b39e4'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-08-24 23:23:18.626495', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-2023aug3-b5', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-08-25T02:17:10.450 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-2023aug3-b5, sha1 ba6b39e4f4f77fb95c747099b53db0467d435f56 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T02:17:10.451 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/x86_64/flavors/default/cephadm 2023-08-25T02:17:10.451 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:11.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Aug 25 02:17 /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:11.162 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.283 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Aug 25 02:17 /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.283 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.891 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Aug 25 02:17 /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.892 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.923 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:12.954 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-08-25T02:17:13.117 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 on all hosts... 2023-08-25T02:17:13.118 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 pull 2023-08-25T02:17:13.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 pull 2023-08-25T02:17:13.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 pull 2023-08-25T02:17:16.036 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56... 2023-08-25T02:17:16.268 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56... 2023-08-25T02:17:16.600 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56... 2023-08-25T02:17:42.613 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:17:42.614 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev)", 2023-08-25T02:17:42.614 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "95c9c69e1344793411bb4e462c28e05fea1317e71809df2f5d3d78facaf76eef", 2023-08-25T02:17:42.614 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2023-08-25T02:17:42.614 INFO:teuthology.orchestra.run.smithi089.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e263fcaadb67f0bae2627ea3f04151ef1bf468f74f698991bba88a3316474934" 2023-08-25T02:17:42.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:17:42.615 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:17:58.993 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-08-25T02:17:58.994 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev)", 2023-08-25T02:17:58.994 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "95c9c69e1344793411bb4e462c28e05fea1317e71809df2f5d3d78facaf76eef", 2023-08-25T02:17:58.994 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2023-08-25T02:17:58.994 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e263fcaadb67f0bae2627ea3f04151ef1bf468f74f698991bba88a3316474934" 2023-08-25T02:17:58.994 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2023-08-25T02:17:58.995 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-08-25T02:18:01.209 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-08-25T02:18:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev)", 2023-08-25T02:18:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "95c9c69e1344793411bb4e462c28e05fea1317e71809df2f5d3d78facaf76eef", 2023-08-25T02:18:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2023-08-25T02:18:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e263fcaadb67f0bae2627ea3f04151ef1bf468f74f698991bba88a3316474934" 2023-08-25T02:18:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-08-25T02:18:01.210 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-08-25T02:18:01.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2023-08-25T02:18:01.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2023-08-25T02:18:01.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2023-08-25T02:18:01.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2023-08-25T02:18:01.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2023-08-25T02:18:01.473 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2023-08-25T02:18:01.515 INFO:tasks.cephadm:Writing seed config... 2023-08-25T02:18:01.573 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-08-25T02:18:01.573 INFO:tasks.cephadm: override: [client] debug client = 20 2023-08-25T02:18:01.574 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-08-25T02:18:01.574 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-08-25T02:18:01.574 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-08-25T02:18:01.574 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-08-25T02:18:01.575 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-08-25T02:18:01.575 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-08-25T02:18:01.575 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-08-25T02:18:01.575 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-08-25T02:18:01.575 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-08-25T02:18:01.576 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-08-25T02:18:01.576 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-08-25T02:18:01.576 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-08-25T02:18:01.576 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-08-25T02:18:01.576 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-08-25T02:18:01.577 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-08-25T02:18:01.577 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-08-25T02:18:01.577 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-08-25T02:18:01.577 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-08-25T02:18:01.577 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-08-25T02:18:01.578 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-08-25T02:18:01.578 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-08-25T02:18:01.578 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-08-25T02:18:01.579 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-08-25T02:18:01.579 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-08-25T02:18:01.579 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-08-25T02:18:01.579 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-08-25T02:18:01.579 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-08-25T02:18:01.580 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-08-25T02:18:01.580 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-08-25T02:18:01.595 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-08-25T02:18:01.595 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-08-25T02:18:01.596 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-08-25T02:18:01.596 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-08-25T02:18:01.596 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-08-25T02:18:01.596 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-08-25T02:18:01.596 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-08-25T02:18:01.597 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-08-25T02:18:01.597 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-08-25T02:18:01.597 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-08-25T02:18:01.597 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-08-25T02:18:01.597 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-08-25T02:18:01.598 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-08-25T02:18:01.598 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-08-25T02:18:01.598 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-08-25T02:18:01.598 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-08-25T02:18:01.600 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:18:01.601 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-08-25T02:18:01.621 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7eb69934-42ed-11ee-9b3d-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-08-25T02:18:01.642 DEBUG:teuthology.orchestra.run.smithi089:mon.a> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service 2023-08-25T02:18:01.679 DEBUG:teuthology.orchestra.run.smithi089:mgr.x> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x.service 2023-08-25T02:18:01.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:18:01.717 INFO:tasks.cephadm:Bootstrapping... 2023-08-25T02:18:01.718 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 -v bootstrap --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.89 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:18:01.737 INFO:journalctl@ceph.mgr.x.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:18:01.880 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-08-25T02:18:01.881 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56', '-v', 'bootstrap', '--fsid', '7eb69934-42ed-11ee-9b3d-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.89', '--skip-admin-label'] 2023-08-25T02:18:02.017 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.1.1 2023-08-25T02:18:02.018 INFO:teuthology.orchestra.run.smithi089.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-08-25T02:18:02.019 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying podman|docker is present... 2023-08-25T02:18:02.169 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.1.1 2023-08-25T02:18:02.170 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying lvm2 is present... 2023-08-25T02:18:02.170 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying time synchronization is in place... 2023-08-25T02:18:02.170 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-08-25T02:18:02.170 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-08-25T02:18:02.190 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-08-25T02:18:02.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:18:02.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2023-08-25T02:18:02.191 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2023-08-25T02:18:02.191 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2023-08-25T02:18:02.192 INFO:teuthology.orchestra.run.smithi089.stdout:Repeating the final host check... 2023-08-25T02:18:02.338 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.1.1 2023-08-25T02:18:02.339 INFO:teuthology.orchestra.run.smithi089.stdout:podman (/bin/podman) version 4.1.1 is present 2023-08-25T02:18:02.340 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl is present 2023-08-25T02:18:02.340 INFO:teuthology.orchestra.run.smithi089.stdout:lvcreate is present 2023-08-25T02:18:02.345 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-08-25T02:18:02.346 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-08-25T02:18:02.383 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-08-25T02:18:02.384 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:18:02.384 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2023-08-25T02:18:02.384 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2023-08-25T02:18:02.384 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2023-08-25T02:18:02.384 INFO:teuthology.orchestra.run.smithi089.stdout:Host looks OK 2023-08-25T02:18:02.413 INFO:teuthology.orchestra.run.smithi089.stdout:Cluster fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:02.413 INFO:teuthology.orchestra.run.smithi089.stdout:Acquiring lock 140420855589352 on /run/cephadm/7eb69934-42ed-11ee-9b3d-001a4aab830c.lock 2023-08-25T02:18:02.413 INFO:teuthology.orchestra.run.smithi089.stdout:Lock 140420855589352 acquired on /run/cephadm/7eb69934-42ed-11ee-9b3d-001a4aab830c.lock 2023-08-25T02:18:02.413 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 3300 ... 2023-08-25T02:18:02.413 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 6789 ... 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:Base mon IP(s) is [172.21.15.89:3300, 172.21.15.89:6789], mon addrv is [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.89 metric 100 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-08-25T02:18:02.414 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-08-25T02:18:02.416 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-08-25T02:18:02.416 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-08-25T02:18:02.416 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 fe80::24ab:d266:e6a3:caa8/64 scope link noprefixroute 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2023-08-25T02:18:02.417 INFO:teuthology.orchestra.run.smithi089.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-08-25T02:18:02.418 INFO:teuthology.orchestra.run.smithi089.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-08-25T02:18:02.418 INFO:teuthology.orchestra.run.smithi089.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56... 2023-08-25T02:18:02.710 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 95c9c69e1344793411bb4e462c28e05fea1317e71809df2f5d3d78facaf76eef 2023-08-25T02:18:02.711 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56... 2023-08-25T02:18:02.711 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Getting image source signatures 2023-08-25T02:18:02.711 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:57753a2dbe9e592c0393092b3155420b9dbb1f83f68296cfecb3d2f029e1ad16 2023-08-25T02:18:02.711 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:6cfdaf403fb5ea185397a3afe33998eb8d2277a1b4936b2369ceaaceb6049ebf 2023-08-25T02:18:02.711 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying config sha256:95c9c69e1344793411bb4e462c28e05fea1317e71809df2f5d3d78facaf76eef 2023-08-25T02:18:02.712 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Writing manifest to image destination 2023-08-25T02:18:02.712 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Storing signatures 2023-08-25T02:18:03.466 INFO:teuthology.orchestra.run.smithi089.stdout:ceph: stdout ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev) 2023-08-25T02:18:03.467 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph version: ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev) 2023-08-25T02:18:03.467 INFO:teuthology.orchestra.run.smithi089.stdout:Extracting ceph user uid/gid from container image... 2023-08-25T02:18:04.118 INFO:teuthology.orchestra.run.smithi089.stdout:stat: stdout 167 167 2023-08-25T02:18:04.118 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial keys... 2023-08-25T02:18:04.767 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQBcD+hkf4LnGBAAobX47116ZFeKxSYlnAzmnQ== 2023-08-25T02:18:05.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQBdD+hkS9i0BhAASXOOFq/weNv9gb38lx3iuQ== 2023-08-25T02:18:06.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQBdD+hk7oxuKxAAsitSahamZCd4vUsT13MRww== 2023-08-25T02:18:06.073 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial monmap... 2023-08-25T02:18:06.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-08-25T02:18:06.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-08-25T02:18:06.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool for a [v2:172.21.15.89:3300,v1:172.21.15.89:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: set fsid to 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:18:06.814 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mon... 2023-08-25T02:18:07.475 INFO:teuthology.orchestra.run.smithi089.stdout:create mon.a on 2023-08-25T02:18:07.585 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-08-25T02:18:07.670 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-08-25T02:18:07.790 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target → /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target. 2023-08-25T02:18:07.790 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target → /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target. 2023-08-25T02:18:07.966 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a 2023-08-25T02:18:07.966 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service: Unit ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service not loaded. 2023-08-25T02:18:08.050 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target.wants/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@.service. 2023-08-25T02:18:08.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 systemd[1]: Starting Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:18:08.555 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-08-25T02:18:08.555 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2023-08-25T02:18:08.563 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-08-25T02:18:08.563 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:18:08.564 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2023-08-25T02:18:08.564 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2023-08-25T02:18:08.564 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon to start... 2023-08-25T02:18:08.564 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon... 2023-08-25T02:18:08.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 podman[98186]: 2023-08-25T02:18:08.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 bash[98186]: 035a4494807b98b287fa1344b975ef902bc1b4a848f57abcf7ee71e6f875dfc7 2023-08-25T02:18:08.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 systemd[1]: Started Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:18:08.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 ceph-mon[98226]: mkfs 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:08.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:08 smithi089 ceph-mon[98226]: mon.a is new leader, mons a in quorum (ranks 0) 2023-08-25T02:18:10.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: mon.a is new leader, mons a in quorum (ranks 0) 2023-08-25T02:18:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: monmap e1: 1 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} 2023-08-25T02:18:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: fsmap 2023-08-25T02:18:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: osdmap e1: 0 total, 0 up, 0 in 2023-08-25T02:18:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: mgrmap e1: no daemons active 2023-08-25T02:18:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:09 smithi089 ceph-mon[98226]: from='client.? 172.21.15.89:0/2906563913' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-08-25T02:18:11.020 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout cluster: 2023-08-25T02:18:11.022 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout id: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:11.022 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-08-25T02:18:11.025 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:11.025 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout services: 2023-08-25T02:18:11.025 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.513864s) 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout data: 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-08-25T02:18:11.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-08-25T02:18:11.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pgs: 2023-08-25T02:18:11.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:11.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.014+0000 7f1bb585d700 1 Processor -- start 2023-08-25T02:18:11.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.014+0000 7f1bb585d700 1 -- start start 2023-08-25T02:18:11.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.014+0000 7f1bb585d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:11.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.014+0000 7f1bb585d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0106cc0 con 0x7f1bb01062d0 2023-08-25T02:18:11.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1baeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:11.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1baeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45338/0 (socket says 172.21.15.89:45338) 2023-08-25T02:18:11.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1baeffd700 1 -- 172.21.15.89:0/2978187343 learned_addr learned my addr 172.21.15.89:0/2978187343 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:11.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1baeffd700 1 -- 172.21.15.89:0/2978187343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb01074d0 con 0x7f1bb01062d0 2023-08-25T02:18:11.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1baeffd700 1 --2- 172.21.15.89:0/2978187343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40098a0 tx=0x7f1ba4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24dc0bd541d735d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:11.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1badffb700 1 -- 172.21.15.89:0/2978187343 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba4004160 con 0x7f1bb01062d0 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1badffb700 1 -- 172.21.15.89:0/2978187343 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba40042c0 con 0x7f1bb01062d0 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.015+0000 7f1badffb700 1 -- 172.21.15.89:0/2978187343 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba40045a0 con 0x7f1bb01062d0 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 -- 172.21.15.89:0/2978187343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 msgr2=0x7f1bb01066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 --2- 172.21.15.89:0/2978187343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40098a0 tx=0x7f1ba4009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 -- 172.21.15.89:0/2978187343 shutdown_connections 2023-08-25T02:18:11.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 --2- 172.21.15.89:0/2978187343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb01066f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 -- 172.21.15.89:0/2978187343 >> 172.21.15.89:0/2978187343 conn(0x7f1bb01018d0 msgr2=0x7f1bb0103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 -- 172.21.15.89:0/2978187343 shutdown_connections 2023-08-25T02:18:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 -- 172.21.15.89:0/2978187343 wait complete. 2023-08-25T02:18:11.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.016+0000 7f1bb585d700 1 Processor -- start 2023-08-25T02:18:11.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb585d700 1 -- start start 2023-08-25T02:18:11.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb585d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:11.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb585d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb007c190 con 0x7f1bb01062d0 2023-08-25T02:18:11.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1baeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:11.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1baeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45340/0 (socket says 172.21.15.89:45340) 2023-08-25T02:18:11.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1baeffd700 1 -- 172.21.15.89:0/2906563913 learned_addr learned my addr 172.21.15.89:0/2906563913 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1baeffd700 1 -- 172.21.15.89:0/2906563913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4009550 con 0x7f1bb01062d0 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1baeffd700 1 --2- 172.21.15.89:0/2906563913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004000 tx=0x7f1ba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba4011630 con 0x7f1bb01062d0 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba4011790 con 0x7f1bb01062d0 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba4011a30 con 0x7f1bb01062d0 2023-08-25T02:18:11.061 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb007a2a0 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.017+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb007a6c0 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.018+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1ba4022070 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.018+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1ba401c450 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.018+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb0061f50 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.021+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f1ba401b450 con 0x7f1bb01062d0 2023-08-25T02:18:11.062 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.090+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1bb0107a00 con 0x7f1bb01062d0 2023-08-25T02:18:11.063 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.090+0000 7f1bb485b700 1 -- 172.21.15.89:0/2906563913 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f1ba401b5f0 con 0x7f1bb01062d0 2023-08-25T02:18:11.063 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 msgr2=0x7f1bb007bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:11.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 --2- 172.21.15.89:0/2906563913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004000 tx=0x7f1ba4004750 comp rx=0 tx=0).stop 2023-08-25T02:18:11.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 shutdown_connections 2023-08-25T02:18:11.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 --2- 172.21.15.89:0/2906563913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb01062d0 0x7f1bb007bc50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:11.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 >> 172.21.15.89:0/2906563913 conn(0x7f1bb01018d0 msgr2=0x7f1bb018dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:11.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 shutdown_connections 2023-08-25T02:18:11.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:09.091+0000 7f1bb585d700 1 -- 172.21.15.89:0/2906563913 wait complete. 2023-08-25T02:18:11.078 INFO:teuthology.orchestra.run.smithi089.stdout:mon is available 2023-08-25T02:18:11.078 INFO:teuthology.orchestra.run.smithi089.stdout:Assimilating anything we can from ceph.conf... 2023-08-25T02:18:13.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:12 smithi089 ceph-mon[98226]: from='client.? 172.21.15.89:0/148235470' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-08-25T02:18:13.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:12 smithi089 ceph-mon[98226]: from='client.? 172.21.15.89:0/148235470' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-08-25T02:18:14.794 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:14.794 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2023-08-25T02:18:14.795 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:14.803 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2023-08-25T02:18:14.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-08-25T02:18:14.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-08-25T02:18:14.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-08-25T02:18:14.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-08-25T02:18:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2023-08-25T02:18:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-08-25T02:18:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:14.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2023-08-25T02:18:14.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.811+0000 7fbf3628f700 1 Processor -- start 2023-08-25T02:18:14.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.811+0000 7fbf3628f700 1 -- start start 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.811+0000 7fbf3628f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.811+0000 7fbf3628f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf301060f0 con 0x7fbf30105700 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45342/0 (socket says 172.21.15.89:45342) 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2ffff700 1 -- 172.21.15.89:0/3379421421 learned_addr learned my addr 172.21.15.89:0/3379421421 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2ffff700 1 -- 172.21.15.89:0/3379421421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30106420 con 0x7fbf30105700 2023-08-25T02:18:14.808 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2ffff700 1 --2- 172.21.15.89:0/3379421421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbf200098a0 tx=0x7fbf20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da17f828ceee270d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2effd700 1 -- 172.21.15.89:0/3379421421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf20004160 con 0x7fbf30105700 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2effd700 1 -- 172.21.15.89:0/3379421421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf200042c0 con 0x7fbf30105700 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.812+0000 7fbf2effd700 1 -- 172.21.15.89:0/3379421421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf200045a0 con 0x7fbf30105700 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 -- 172.21.15.89:0/3379421421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 msgr2=0x7fbf30105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 --2- 172.21.15.89:0/3379421421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbf200098a0 tx=0x7fbf20009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 -- 172.21.15.89:0/3379421421 shutdown_connections 2023-08-25T02:18:14.809 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 --2- 172.21.15.89:0/3379421421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30105b20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:14.810 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 -- 172.21.15.89:0/3379421421 >> 172.21.15.89:0/3379421421 conn(0x7fbf300fff00 msgr2=0x7fbf30102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:14.810 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 -- 172.21.15.89:0/3379421421 shutdown_connections 2023-08-25T02:18:14.810 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.813+0000 7fbf3628f700 1 -- 172.21.15.89:0/3379421421 wait complete. 2023-08-25T02:18:14.810 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf3628f700 1 Processor -- start 2023-08-25T02:18:14.811 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf3628f700 1 -- start start 2023-08-25T02:18:14.811 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf3628f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:14.811 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf3628f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3019c4e0 con 0x7fbf30105700 2023-08-25T02:18:14.812 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:14.812 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45344/0 (socket says 172.21.15.89:45344) 2023-08-25T02:18:14.812 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf2ffff700 1 -- 172.21.15.89:0/148235470 learned_addr learned my addr 172.21.15.89:0/148235470 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:14.812 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.814+0000 7fbf2ffff700 1 -- 172.21.15.89:0/148235470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf20009550 con 0x7fbf30105700 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2ffff700 1 --2- 172.21.15.89:0/148235470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbf300665d0 tx=0x7fbf20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf20011630 con 0x7fbf30105700 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20011790 con 0x7fbf30105700 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3019c620 con 0x7fbf30105700 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf20011a30 con 0x7fbf30105700 2023-08-25T02:18:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3019ca40 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbf20022070 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.815+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbf2001c450 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.816+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf30191f40 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.818+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fbf2001b450 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.884+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fbf30106790 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.888+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf2001b5f0 con 0x7fbf30105700 2023-08-25T02:18:14.814 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.888+0000 7fbf2d7fa700 1 -- 172.21.15.89:0/148235470 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fbf20024630 con 0x7fbf30105700 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 msgr2=0x7fbf30104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 --2- 172.21.15.89:0/148235470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbf300665d0 tx=0x7fbf20004750 comp rx=0 tx=0).stop 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 shutdown_connections 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 --2- 172.21.15.89:0/148235470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf30105700 0x7fbf30104dc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 >> 172.21.15.89:0/148235470 conn(0x7fbf300fff00 msgr2=0x7fbf3018cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 shutdown_connections 2023-08-25T02:18:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:12.889+0000 7fbf3628f700 1 -- 172.21.15.89:0/148235470 wait complete. 2023-08-25T02:18:14.816 INFO:teuthology.orchestra.run.smithi089.stdout:Generating new minimal ceph.conf... 2023-08-25T02:18:17.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:17 smithi089 ceph-mon[98226]: from='client.? 172.21.15.89:0/1767815945' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:18:18.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.039+0000 7f5ebdeb7700 1 Processor -- start 2023-08-25T02:18:18.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.039+0000 7f5ebdeb7700 1 -- start start 2023-08-25T02:18:18.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5ebdeb7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:18.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5ebdeb7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8106cd0 con 0x7f5eb81062e0 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45346/0 (socket says 172.21.15.89:45346) 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5eb77fe700 1 -- 172.21.15.89:0/2041588544 learned_addr learned my addr 172.21.15.89:0/2041588544 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5eb77fe700 1 -- 172.21.15.89:0/2041588544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb81074e0 con 0x7f5eb81062e0 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.040+0000 7f5eb77fe700 1 --2- 172.21.15.89:0/2041588544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00098a0 tx=0x7f5ea0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9031112649f80814 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5eb67fc700 1 -- 172.21.15.89:0/2041588544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea0004160 con 0x7f5eb81062e0 2023-08-25T02:18:18.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5eb67fc700 1 -- 172.21.15.89:0/2041588544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ea00042c0 con 0x7f5eb81062e0 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5eb67fc700 1 -- 172.21.15.89:0/2041588544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea0011a00 con 0x7f5eb81062e0 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/2041588544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 msgr2=0x7f5eb8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 --2- 172.21.15.89:0/2041588544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00098a0 tx=0x7f5ea0009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/2041588544 shutdown_connections 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 --2- 172.21.15.89:0/2041588544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8106700 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/2041588544 >> 172.21.15.89:0/2041588544 conn(0x7f5eb8101860 msgr2=0x7f5eb8103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:18.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/2041588544 shutdown_connections 2023-08-25T02:18:18.660 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.041+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/2041588544 wait complete. 2023-08-25T02:18:18.660 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5ebdeb7700 1 Processor -- start 2023-08-25T02:18:18.660 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5ebdeb7700 1 -- start start 2023-08-25T02:18:18.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5ebdeb7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:18.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5ebdeb7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb819a4d0 con 0x7f5eb81062e0 2023-08-25T02:18:18.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:18.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45348/0 (socket says 172.21.15.89:45348) 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.042+0000 7f5eb77fe700 1 -- 172.21.15.89:0/1767815945 learned_addr learned my addr 172.21.15.89:0/1767815945 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb77fe700 1 -- 172.21.15.89:0/1767815945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0009550 con 0x7f5eb81062e0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb77fe700 1 --2- 172.21.15.89:0/1767815945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006af0 tx=0x7f5ea0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea0003a40 con 0x7f5eb81062e0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ea0003ba0 con 0x7f5eb81062e0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb819a6d0 con 0x7f5eb81062e0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea001a700 con 0x7f5eb81062e0 2023-08-25T02:18:18.666 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb819aaf0 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5ea001ac30 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.043+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ea0019700 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.044+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb8061f50 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.046+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f5ea002de40 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.113+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5eb810a6e0 con 0x7f5eb81062e0 2023-08-25T02:18:18.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.113+0000 7f5eb4ff9700 1 -- 172.21.15.89:0/1767815945 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f5ea0024ad0 con 0x7f5eb81062e0 2023-08-25T02:18:18.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 msgr2=0x7f5eb8199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:18.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 --2- 172.21.15.89:0/1767815945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006af0 tx=0x7f5ea0011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:18.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 shutdown_connections 2023-08-25T02:18:18.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 --2- 172.21.15.89:0/1767815945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eb81062e0 0x7f5eb8199f90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:18.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 >> 172.21.15.89:0/1767815945 conn(0x7f5eb8101860 msgr2=0x7f5eb8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:18.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 shutdown_connections 2023-08-25T02:18:18.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:17.114+0000 7f5ebdeb7700 1 -- 172.21.15.89:0/1767815945 wait complete. 2023-08-25T02:18:18.669 INFO:teuthology.orchestra.run.smithi089.stdout:Restarting the monitor... 2023-08-25T02:18:18.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:18 smithi089 systemd[1]: Stopping Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:18:19.245 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a[98202]: 2023-08-25T02:18:18.987+0000 7fd2b00cf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:18:19.245 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a[98202]: 2023-08-25T02:18:18.987+0000 7fd2b00cf700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-08-25T02:18:19.245 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 bash[98560]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a 2023-08-25T02:18:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 systemd[1]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service: Succeeded. 2023-08-25T02:18:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 systemd[1]: Stopped Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:18:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 systemd[1]: Starting Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:18:20.006 INFO:teuthology.orchestra.run.smithi089.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-08-25T02:18:20.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 podman[98693]: 2023-08-25T02:18:20.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 bash[98693]: 4e5b0a3428c19c15745c06caf0e9920023b3c85bd877b9904f55e7b084400b60 2023-08-25T02:18:20.087 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: set uid:gid to 167:167 (ceph:ceph) 2023-08-25T02:18:20.087 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev), process ceph-mon, pid 7 2023-08-25T02:18:20.092 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: pidfile_write: ignore empty --pid-file 2023-08-25T02:18:20.095 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: load: jerasure load: lrc 2023-08-25T02:18:20.103 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: RocksDB version: 7.9.2 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Git sha 0 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Compile date 2023-08-24 22:08:31 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: DB SUMMARY 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: DB Session ID: RVI5570DXI6YBXH5Q86Q 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: CURRENT file: CURRENT 2023-08-25T02:18:20.104 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: IDENTITY file: IDENTITY 2023-08-25T02:18:20.125 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-08-25T02:18:20.127 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-08-25T02:18:20.128 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 101133 ; 2023-08-25T02:18:20.128 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.error_if_exists: 0 2023-08-25T02:18:20.128 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.create_if_missing: 0 2023-08-25T02:18:20.136 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.paranoid_checks: 1 2023-08-25T02:18:20.137 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.flush_verify_memtable_count: 1 2023-08-25T02:18:20.170 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-08-25T02:18:20.170 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-08-25T02:18:20.171 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.env: 0x560ebc992da0 2023-08-25T02:18:20.171 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.fs: PosixFileSystem 2023-08-25T02:18:20.171 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.info_log: 0x560ebecb5440 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_file_opening_threads: 16 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.statistics: (nil) 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.use_fsync: 0 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_log_file_size: 0 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-08-25T02:18:20.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.log_file_time_to_roll: 0 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.keep_log_file_num: 1000 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.recycle_log_file_num: 0 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_fallocate: 1 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_mmap_reads: 0 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_mmap_writes: 0 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.use_direct_reads: 0 2023-08-25T02:18:20.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.create_missing_column_families: 0 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.db_log_dir: 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.wal_dir: 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.table_cache_numshardbits: 6 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.WAL_ttl_seconds: 0 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.WAL_size_limit_MB: 0 2023-08-25T02:18:20.174 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.is_fd_close_on_exec: 1 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.advise_random_on_open: 1 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.db_write_buffer_size: 0 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.write_buffer_manager: 0x560ebec6f4a0 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-08-25T02:18:20.175 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-08-25T02:18:20.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.use_adaptive_mutex: 0 2023-08-25T02:18:20.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.rate_limiter: (nil) 2023-08-25T02:18:20.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-08-25T02:18:20.176 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.wal_recovery_mode: 2 2023-08-25T02:18:20.177 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enable_thread_tracking: 0 2023-08-25T02:18:20.177 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enable_pipelined_write: 0 2023-08-25T02:18:20.177 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.unordered_write: 0 2023-08-25T02:18:20.177 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-08-25T02:18:20.178 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-08-25T02:18:20.178 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-08-25T02:18:20.178 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-08-25T02:18:20.178 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.row_cache: None 2023-08-25T02:18:20.178 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.wal_filter: None 2023-08-25T02:18:20.179 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-08-25T02:18:20.179 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_ingest_behind: 0 2023-08-25T02:18:20.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.two_write_queues: 0 2023-08-25T02:18:20.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.manual_wal_flush: 0 2023-08-25T02:18:20.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.wal_compression: 0 2023-08-25T02:18:20.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.atomic_flush: 0 2023-08-25T02:18:20.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-08-25T02:18:20.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.persist_stats_to_disk: 0 2023-08-25T02:18:20.193 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.write_dbid_to_manifest: 0 2023-08-25T02:18:20.193 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.log_readahead_size: 0 2023-08-25T02:18:20.193 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-08-25T02:18:20.194 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.best_efforts_recovery: 0 2023-08-25T02:18:20.194 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-08-25T02:18:20.194 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-08-25T02:18:20.194 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.allow_data_in_errors: 0 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.db_host_id: __hostname__ 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enforce_single_del_contracts: true 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_background_jobs: 2 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_background_compactions: -1 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_subcompactions: 1 2023-08-25T02:18:20.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-08-25T02:18:20.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-08-25T02:18:20.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.delayed_write_rate : 16777216 2023-08-25T02:18:20.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_total_wal_size: 0 2023-08-25T02:18:20.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-08-25T02:18:20.196 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.stats_dump_period_sec: 600 2023-08-25T02:18:20.209 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.stats_persist_period_sec: 600 2023-08-25T02:18:20.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-08-25T02:18:20.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_open_files: -1 2023-08-25T02:18:20.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bytes_per_sync: 0 2023-08-25T02:18:20.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.wal_bytes_per_sync: 0 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.strict_bytes_per_sync: 0 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_readahead_size: 0 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_background_flushes: -1 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Compression algorithms supported: 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kZSTD supported: 0 2023-08-25T02:18:20.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kXpressCompression supported: 0 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kLZ4HCCompression supported: 1 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kZlibCompression supported: 1 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kSnappyCompression supported: 1 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kLZ4Compression supported: 1 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: kBZip2Compression supported: 0 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Fast CRC32 supported: Supported on x86 2023-08-25T02:18:20.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: DMutex implementation: pthread_mutex_t 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.merge_operator: 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_filter: None 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_filter_factory: None 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.sst_partitioner_factory: None 2023-08-25T02:18:20.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.memtable_factory: SkipListFactory 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.table_factory: BlockBasedTable 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560ebecb5540) 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks: 1 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_top_level_index_and_filter: 1 2023-08-25T02:18:20.219 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_type: 0 2023-08-25T02:18:20.220 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_index_type: 0 2023-08-25T02:18:20.220 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_shortening: 1 2023-08-25T02:18:20.221 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_hash_table_util_ratio: 0.750000 2023-08-25T02:18:20.221 INFO:journalctl@ceph.mon.a.smithi089.stdout: checksum: 4 2023-08-25T02:18:20.221 INFO:journalctl@ceph.mon.a.smithi089.stdout: no_block_cache: 0 2023-08-25T02:18:20.223 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache: 0x560ebed0d1f0 2023-08-25T02:18:20.223 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_name: BinnedLRUCache 2023-08-25T02:18:20.223 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_options: 2023-08-25T02:18:20.223 INFO:journalctl@ceph.mon.a.smithi089.stdout: capacity : 536870912 2023-08-25T02:18:20.223 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_shard_bits : 4 2023-08-25T02:18:20.224 INFO:journalctl@ceph.mon.a.smithi089.stdout: strict_capacity_limit : 0 2023-08-25T02:18:20.224 INFO:journalctl@ceph.mon.a.smithi089.stdout: high_pri_pool_ratio: 0.000 2023-08-25T02:18:20.224 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_compressed: (nil) 2023-08-25T02:18:20.225 INFO:journalctl@ceph.mon.a.smithi089.stdout: persistent_cache: (nil) 2023-08-25T02:18:20.225 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size: 4096 2023-08-25T02:18:20.226 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size_deviation: 10 2023-08-25T02:18:20.226 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_restart_interval: 16 2023-08-25T02:18:20.226 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_block_restart_interval: 1 2023-08-25T02:18:20.226 INFO:journalctl@ceph.mon.a.smithi089.stdout: metadata_block_size: 4096 2023-08-25T02:18:20.226 INFO:journalctl@ceph.mon.a.smithi089.stdout: partition_filters: 0 2023-08-25T02:18:20.227 INFO:journalctl@ceph.mon.a.smithi089.stdout: use_delta_encoding: 1 2023-08-25T02:18:20.227 INFO:journalctl@ceph.mon.a.smithi089.stdout: filter_policy: bloomfilter 2023-08-25T02:18:20.227 INFO:journalctl@ceph.mon.a.smithi089.stdout: whole_key_filtering: 1 2023-08-25T02:18:20.227 INFO:journalctl@ceph.mon.a.smithi089.stdout: verify_compression: 0 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: read_amp_bytes_per_bit: 0 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: format_version: 5 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_index_compression: 1 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_align: 0 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: max_auto_readahead_size: 262144 2023-08-25T02:18:20.228 INFO:journalctl@ceph.mon.a.smithi089.stdout: prepopulate_block_cache: 0 2023-08-25T02:18:20.229 INFO:journalctl@ceph.mon.a.smithi089.stdout: initial_auto_readahead_size: 8192 2023-08-25T02:18:20.229 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_file_reads_for_auto_readahead: 2 2023-08-25T02:18:20.229 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.write_buffer_size: 33554432 2023-08-25T02:18:20.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_write_buffer_number: 2 2023-08-25T02:18:20.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression: NoCompression 2023-08-25T02:18:20.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression: Disabled 2023-08-25T02:18:20.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.prefix_extractor: nullptr 2023-08-25T02:18:20.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.num_levels: 7 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-08-25T02:18:20.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-08-25T02:18:20.232 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-08-25T02:18:20.232 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:18:20.240 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.window_bits: -14 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.level: 32767 2023-08-25T02:18:20.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.strategy: 0 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.enabled: false 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-08-25T02:18:20.242 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-08-25T02:18:20.243 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-08-25T02:18:20.260 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.target_file_size_base: 67108864 2023-08-25T02:18:20.260 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.target_file_size_multiplier: 1 2023-08-25T02:18:20.260 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-08-25T02:18:20.270 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-08-25T02:18:20.270 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-08-25T02:18:20.270 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-08-25T02:18:20.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-08-25T02:18:20.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-08-25T02:18:20.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-08-25T02:18:20.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-08-25T02:18:20.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-08-25T02:18:20.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-08-25T02:18:20.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-08-25T02:18:20.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-08-25T02:18:20.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-08-25T02:18:20.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.arena_block_size: 1048576 2023-08-25T02:18:20.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-08-25T02:18:20.274 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-08-25T02:18:20.275 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.disable_auto_compactions: 0 2023-08-25T02:18:20.275 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-08-25T02:18:20.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-08-25T02:18:20.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-08-25T02:18:20.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-08-25T02:18:20.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-08-25T02:18:20.281 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-08-25T02:18:20.281 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-08-25T02:18:20.281 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-08-25T02:18:20.281 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-08-25T02:18:20.282 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.table_properties_collectors: 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.inplace_update_support: 0 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.inplace_update_num_locks: 10000 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-08-25T02:18:20.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.memtable_huge_page_size: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.bloom_locality: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.max_successive_merges: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.optimize_filters_for_hits: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.paranoid_file_checks: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.force_consistency_checks: 1 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.report_bg_io_stats: 0 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.ttl: 2592000 2023-08-25T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.periodic_compaction_seconds: 0 2023-08-25T02:18:20.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-08-25T02:18:20.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-08-25T02:18:20.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enable_blob_files: false 2023-08-25T02:18:20.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.min_blob_size: 0 2023-08-25T02:18:20.286 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_file_size: 268435456 2023-08-25T02:18:20.286 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_compression_type: NoCompression 2023-08-25T02:18:20.286 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.enable_blob_garbage_collection: false 2023-08-25T02:18:20.286 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-08-25T02:18:20.287 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-08-25T02:18:20.287 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-08-25T02:18:20.287 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.blob_file_starting_level: 0 2023-08-25T02:18:20.287 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 83db1d62-e071-49ac-ac18-d8cfee9f437c 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929899993044, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-08-25T02:18:20.289 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929899994051, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95757, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 338, "table_properties": {"data_size": 93745, "index_size": 294, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 14023, "raw_average_key_size": 49, "raw_value_size": 86109, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1692929899, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "83db1d62-e071-49ac-ac18-d8cfee9f437c", "db_session_id": "RVI5570DXI6YBXH5Q86Q", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-08-25T02:18:20.290 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929899994119, "job": 1, "event": "recovery_finished"} 2023-08-25T02:18:20.290 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560ebedaa000 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: DB pointer 0x560ebed96000 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-08-25T02:18:20.291 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** DB Stats ** 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-08-25T02:18:20.292 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:18:20.301 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:18:20.305 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.305 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2023-08-25T02:18:20.315 INFO:journalctl@ceph.mon.a.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:18:20.315 INFO:journalctl@ceph.mon.a.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-08-25T02:18:20.315 INFO:journalctl@ceph.mon.a.smithi089.stdout: L0 2/0 95.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-08-25T02:18:20.316 INFO:journalctl@ceph.mon.a.smithi089.stdout: Sum 2/0 95.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-08-25T02:18:20.316 INFO:journalctl@ceph.mon.a.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-08-25T02:18:20.319 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.319 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2023-08-25T02:18:20.320 INFO:journalctl@ceph.mon.a.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:18:20.320 INFO:journalctl@ceph.mon.a.smithi089.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-08-25T02:18:20.320 INFO:journalctl@ceph.mon.a.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-08-25T02:18:20.321 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.321 INFO:journalctl@ceph.mon.a.smithi089.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-08-25T02:18:20.321 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.321 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:18:20.321 INFO:journalctl@ceph.mon.a.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-08-25T02:18:20.322 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-08-25T02:18:20.322 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-08-25T02:18:20.322 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-08-25T02:18:20.322 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2023-08-25T02:18:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative compaction: 0.00 GB write, 31.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:18:20.325 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval compaction: 0.00 GB write, 31.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:18:20.326 INFO:journalctl@ceph.mon.a.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-08-25T02:18:20.326 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache BinnedLRUCache@0x560ebed0d1f0#7 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-08-25T02:18:20.326 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2023-08-25T02:18:20.338 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.338 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2023-08-25T02:18:20.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: starting mon.a rank 0 at public addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] at bind addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:20.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???) e1 preinit fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:20.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).mds e1 new map 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).mds e1 print_map 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout: e1 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout: legacy client fscid: -1 2023-08-25T02:18:20.339 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2023-08-25T02:18:20.340 INFO:journalctl@ceph.mon.a.smithi089.stdout: No filesystems configured 2023-08-25T02:18:20.340 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expand map: {default=false} 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta from 'false' to 'false' 2023-08-25T02:18:20.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expanded map: {default=false} 2023-08-25T02:18:20.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expand map: {default=info} 2023-08-25T02:18:20.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta from 'info' to 'info' 2023-08-25T02:18:20.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expanded map: {default=info} 2023-08-25T02:18:20.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expand map: {default=daemon} 2023-08-25T02:18:20.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta from 'daemon' to 'daemon' 2023-08-25T02:18:20.344 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expanded map: {default=daemon} 2023-08-25T02:18:20.344 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expand map: {default=debug} 2023-08-25T02:18:20.345 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta from 'debug' to 'debug' 2023-08-25T02:18:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:19 smithi089 ceph-mon[98732]: expand_channel_meta expanded map: {default=debug} 2023-08-25T02:18:20.348 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 systemd[1]: Started Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:18:20.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 ceph-mon[98732]: mon.a is new leader, mons a in quorum (ranks 0) 2023-08-25T02:18:20.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 ceph-mon[98732]: monmap e1: 1 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} 2023-08-25T02:18:20.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 ceph-mon[98732]: fsmap 2023-08-25T02:18:20.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 ceph-mon[98732]: osdmap e1: 0 total, 0 up, 0 in 2023-08-25T02:18:20.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:20 smithi089 ceph-mon[98732]: mgrmap e1: no daemons active 2023-08-25T02:18:20.859 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.474+0000 7f1f80f19700 1 Processor -- start 2023-08-25T02:18:20.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.474+0000 7f1f80f19700 1 -- start start 2023-08-25T02:18:20.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.474+0000 7f1f80f19700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:20.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.474+0000 7f1f80f19700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c104fb0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45350/0 (socket says 172.21.15.89:45350) 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7a59c700 1 -- 172.21.15.89:0/636839799 learned_addr learned my addr 172.21.15.89:0/636839799 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7a59c700 1 -- 172.21.15.89:0/636839799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c1050f0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7a59c700 1 --2- 172.21.15.89:0/636839799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0098a0 tx=0x7f1f6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9469ee691ba10921 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7959a700 1 -- 172.21.15.89:0/636839799 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c004160 con 0x7f1f7c1045c0 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7959a700 1 -- 172.21.15.89:0/636839799 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c0042c0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.475+0000 7f1f7959a700 1 -- 172.21.15.89:0/636839799 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c011a00 con 0x7f1f7c1045c0 2023-08-25T02:18:20.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 -- 172.21.15.89:0/636839799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 msgr2=0x7f1f7c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:20.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 --2- 172.21.15.89:0/636839799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0098a0 tx=0x7f1f6c009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:20.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 -- 172.21.15.89:0/636839799 shutdown_connections 2023-08-25T02:18:20.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 --2- 172.21.15.89:0/636839799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c1049e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:20.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 -- 172.21.15.89:0/636839799 >> 172.21.15.89:0/636839799 conn(0x7f1f7c0fff00 msgr2=0x7f1f7c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:20.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 -- 172.21.15.89:0/636839799 shutdown_connections 2023-08-25T02:18:20.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.476+0000 7f1f80f19700 1 -- 172.21.15.89:0/636839799 wait complete. 2023-08-25T02:18:20.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 Processor -- start 2023-08-25T02:18:20.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 -- start start 2023-08-25T02:18:20.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:20.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1074e0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f7a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:20.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f7a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45352/0 (socket says 172.21.15.89:45352) 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f7a59c700 1 -- 172.21.15.89:0/2000468401 learned_addr learned my addr 172.21.15.89:0/2000468401 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f7a59c700 1 -- 172.21.15.89:0/2000468401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c009550 con 0x7f1f7c1045c0 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f7a59c700 1 --2- 172.21.15.89:0/2000468401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fb0 tx=0x7f1f6c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c003a40 con 0x7f1f7c1045c0 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c003ba0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f6c01a750 con 0x7f1f7c1045c0 2023-08-25T02:18:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c1055f0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.477+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c105a10 con 0x7f1f7c1045c0 2023-08-25T02:18:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.478+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1f6c01ac80 con 0x7f1f7c1045c0 2023-08-25T02:18:20.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.478+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1f6c0193c0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.478+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c061f50 con 0x7f1f7c1045c0 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.481+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f1f6c02de30 con 0x7f1f7c1045c0 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.546+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1f7c193cb0 con 0x7f1f7c1045c0 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.548+0000 7f1f6b7fe700 1 -- 172.21.15.89:0/2000468401 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1f6c024b60 con 0x7f1f7c1045c0 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 msgr2=0x7f1f7c106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:20.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 --2- 172.21.15.89:0/2000468401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fb0 tx=0x7f1f6c011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 shutdown_connections 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 --2- 172.21.15.89:0/2000468401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f7c1045c0 0x7f1f7c106fa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 >> 172.21.15.89:0/2000468401 conn(0x7f1f7c0fff00 msgr2=0x7f1f7c101690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 shutdown_connections 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:20.549+0000 7f1f80f19700 1 -- 172.21.15.89:0/2000468401 wait complete. 2023-08-25T02:18:20.892 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote config to /etc/ceph/ceph.conf 2023-08-25T02:18:20.910 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:18:20.914 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mgr... 2023-08-25T02:18:20.914 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:9283 ... 2023-08-25T02:18:20.914 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:8765 ... 2023-08-25T02:18:21.068 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x 2023-08-25T02:18:21.068 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x.service: Unit ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x.service not loaded. 2023-08-25T02:18:21.157 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c.target.wants/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@.service. 2023-08-25T02:18:21.456 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 systemd[1]: Starting Ceph mgr.x for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:18:21.713 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:21 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2000468401' entity='client.admin' 2023-08-25T02:18:21.739 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 podman[98961]: 2023-08-25T02:18:21.765 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-08-25T02:18:21.765 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2023-08-25T02:18:21.766 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-08-25T02:18:21.766 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:18:21.766 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2023-08-25T02:18:21.766 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2023-08-25T02:18:21.778 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-08-25T02:18:21.781 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr to start... 2023-08-25T02:18:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr... 2023-08-25T02:18:21.970 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 bash[98961]: 90e8ce7beaf35d4355988b52bda9696defa5f70c35f06857fe140c0117522319 2023-08-25T02:18:21.971 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 systemd[1]: Started Ceph mgr.x for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:18:21.971 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:21.844+0000 7fb1a9dee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:18:21.971 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:21.926+0000 7fb1a9dee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:18:22.309 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.022+0000 7fb1a9dee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:18:22.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.118+0000 7fb1a9dee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:18:22.569 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.297+0000 7fb1a9dee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:18:22.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-08-25T02:18:22.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2023-08-25T02:18:22.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:22.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-08-25T02:18:22.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2023-08-25T02:18:22.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:22.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-08-25T02:18:22.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-08-25T02:18:22.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-08-25T02:18:22.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-08-25T02:18:22.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:22.663 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-08-25T02:18:22.663 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2023-08-25T02:18:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2023-08-25T02:18:22.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2023-08-25T02:18:22.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2023-08-25T02:18:22.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:22.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:22.665 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2023-08-25T02:18:08.578980+0000", 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:22.667 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.243+0000 7f9403bfb700 1 Processor -- start 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.243+0000 7f9403bfb700 1 -- start start 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.243+0000 7f9403bfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.243+0000 7f9403bfb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc1060f0 con 0x7f93fc105700 2023-08-25T02:18:22.668 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9401997700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9401997700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45358/0 (socket says 172.21.15.89:45358) 2023-08-25T02:18:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9401997700 1 -- 172.21.15.89:0/3159219182 learned_addr learned my addr 172.21.15.89:0/3159219182 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9401997700 1 -- 172.21.15.89:0/3159219182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc106420 con 0x7f93fc105700 2023-08-25T02:18:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9401997700 1 --2- 172.21.15.89:0/3159219182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00bb40 tx=0x7f93ec00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65dbe49177883749 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:22.669 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9400995700 1 -- 172.21.15.89:0/3159219182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec004160 con 0x7f93fc105700 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9400995700 1 -- 172.21.15.89:0/3159219182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93ec0042c0 con 0x7f93fc105700 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.244+0000 7f9400995700 1 -- 172.21.15.89:0/3159219182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec003a50 con 0x7f93fc105700 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 -- 172.21.15.89:0/3159219182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 msgr2=0x7f93fc105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 --2- 172.21.15.89:0/3159219182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00bb40 tx=0x7f93ec00b4d0 comp rx=0 tx=0).stop 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 -- 172.21.15.89:0/3159219182 shutdown_connections 2023-08-25T02:18:22.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 --2- 172.21.15.89:0/3159219182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc105b20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:22.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 -- 172.21.15.89:0/3159219182 >> 172.21.15.89:0/3159219182 conn(0x7f93fc0fff00 msgr2=0x7f93fc102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:22.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 -- 172.21.15.89:0/3159219182 shutdown_connections 2023-08-25T02:18:22.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.245+0000 7f9403bfb700 1 -- 172.21.15.89:0/3159219182 wait complete. 2023-08-25T02:18:22.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9403bfb700 1 Processor -- start 2023-08-25T02:18:22.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9403bfb700 1 -- start start 2023-08-25T02:18:22.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9403bfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:22.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9403bfb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc1982f0 con 0x7f93fc105700 2023-08-25T02:18:22.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9401997700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:22.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9401997700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45360/0 (socket says 172.21.15.89:45360) 2023-08-25T02:18:22.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.246+0000 7f9401997700 1 -- 172.21.15.89:0/3410592244 learned_addr learned my addr 172.21.15.89:0/3410592244 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:22.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f9401997700 1 -- 172.21.15.89:0/3410592244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec00b7f0 con 0x7f93fc105700 2023-08-25T02:18:22.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f9401997700 1 --2- 172.21.15.89:0/3410592244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f93ec006fa0 tx=0x7f93ec004100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:22.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec0118f0 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93ec011a50 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93fc1984f0 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec0195b0 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc198910 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f93ec019760 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.247+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f93ec015070 con 0x7f93fc105700 2023-08-25T02:18:22.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.248+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc191ab0 con 0x7f93fc105700 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.250+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f93ec027020 con 0x7f93fc105700 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.323+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f93fc061fa0 con 0x7f93fc105700 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.323+0000 7f93f2ffd700 1 -- 172.21.15.89:0/3410592244 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f93ec01e040 con 0x7f93fc105700 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.324+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 msgr2=0x7f93fc197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.324+0000 7f9403bfb700 1 --2- 172.21.15.89:0/3410592244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f93ec006fa0 tx=0x7f93ec004100 comp rx=0 tx=0).stop 2023-08-25T02:18:22.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.324+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 shutdown_connections 2023-08-25T02:18:22.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.324+0000 7f9403bfb700 1 --2- 172.21.15.89:0/3410592244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93fc105700 0x7f93fc197db0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:22.709 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.324+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 >> 172.21.15.89:0/3410592244 conn(0x7f93fc0fff00 msgr2=0x7f93fc18ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:22.710 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.325+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 shutdown_connections 2023-08-25T02:18:22.710 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:22.325+0000 7f9403bfb700 1 -- 172.21.15.89:0/3410592244 wait complete. 2023-08-25T02:18:22.710 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (1/15)... 2023-08-25T02:18:22.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3410592244' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-08-25T02:18:22.819 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.685+0000 7fb1a9dee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:18:23.081 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.814+0000 7fb1a9dee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:18:23.088 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.902+0000 7fb1a9dee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:18:23.088 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:22.996+0000 7fb1a9dee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:18:23.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:23.410+0000 7fb1a9dee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:18:24.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:24.066+0000 7fb1a9dee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:18:24.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:24.147+0000 7fb1a9dee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:18:25.196 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:24.923+0000 7fb1a9dee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:18:25.197 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:25.016+0000 7fb1a9dee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:18:25.493 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:25.194+0000 7fb1a9dee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:18:25.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:25.614 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:25.615 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:18:25.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2023-08-25T02:18:25.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-08-25T02:18:25.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-08-25T02:18:25.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-08-25T02:18:25.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-08-25T02:18:25.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2023-08-25T02:18:25.629 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:25.629 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-08-25T02:18:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-08-25T02:18:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-08-25T02:18:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-08-25T02:18:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-08-25T02:18:25.636 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-08-25T02:18:25.636 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-08-25T02:18:25.636 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-08-25T02:18:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2023-08-25T02:18:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-08-25T02:18:25.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2023-08-25T02:18:25.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2023-08-25T02:18:25.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2023-08-25T02:18:25.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2023-08-25T02:18:08.578980+0000", 2023-08-25T02:18:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.123+0000 7fc7432b9700 1 Processor -- start 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.123+0000 7fc7432b9700 1 -- start start 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc7432b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:25.649 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc7432b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c104fb0 con 0x7fc73c1045c0 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc741055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc741055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45362/0 (socket says 172.21.15.89:45362) 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc741055700 1 -- 172.21.15.89:0/3295014317 learned_addr learned my addr 172.21.15.89:0/3295014317 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc741055700 1 -- 172.21.15.89:0/3295014317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c1050f0 con 0x7fc73c1045c0 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc741055700 1 --2- 172.21.15.89:0/3295014317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098a0 tx=0x7fc738009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b1f60d585b11c4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.124+0000 7fc733fff700 1 -- 172.21.15.89:0/3295014317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc738004160 con 0x7fc73c1045c0 2023-08-25T02:18:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc733fff700 1 -- 172.21.15.89:0/3295014317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7380042c0 con 0x7fc73c1045c0 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc733fff700 1 -- 172.21.15.89:0/3295014317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc738011a00 con 0x7fc73c1045c0 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 -- 172.21.15.89:0/3295014317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 msgr2=0x7fc73c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 --2- 172.21.15.89:0/3295014317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098a0 tx=0x7fc738009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 -- 172.21.15.89:0/3295014317 shutdown_connections 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 --2- 172.21.15.89:0/3295014317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c1049e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 -- 172.21.15.89:0/3295014317 >> 172.21.15.89:0/3295014317 conn(0x7fc73c0fff00 msgr2=0x7fc73c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 -- 172.21.15.89:0/3295014317 shutdown_connections 2023-08-25T02:18:25.651 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.125+0000 7fc7432b9700 1 -- 172.21.15.89:0/3295014317 wait complete. 2023-08-25T02:18:25.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc7432b9700 1 Processor -- start 2023-08-25T02:18:25.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc7432b9700 1 -- start start 2023-08-25T02:18:25.652 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc7432b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:25.653 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc7432b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c19a550 con 0x7fc73c1045c0 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc741055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc741055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45364/0 (socket says 172.21.15.89:45364) 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc741055700 1 -- 172.21.15.89:0/2363091907 learned_addr learned my addr 172.21.15.89:0/2363091907 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.126+0000 7fc741055700 1 -- 172.21.15.89:0/2363091907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738009550 con 0x7fc73c1045c0 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc741055700 1 --2- 172.21.15.89:0/2363091907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc738009fb0 tx=0x7fc738011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:25.654 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc738003a40 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc738003ba0 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c19a750 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc73801a750 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c19ab70 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc73801ac80 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.127+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc7380193c0 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.128+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c1940e0 con 0x7fc73c1045c0 2023-08-25T02:18:25.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.130+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fc738024b60 con 0x7fc73c1045c0 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.202+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc73c02cbd0 con 0x7fc73c1045c0 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.202+0000 7fc7327fc700 1 -- 172.21.15.89:0/2363091907 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fc738024d00 con 0x7fc73c1045c0 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 msgr2=0x7fc73c19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 --2- 172.21.15.89:0/2363091907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc738009fb0 tx=0x7fc738011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 shutdown_connections 2023-08-25T02:18:25.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 --2- 172.21.15.89:0/2363091907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc73c1045c0 0x7fc73c19a010 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:25.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 >> 172.21.15.89:0/2363091907 conn(0x7fc73c0fff00 msgr2=0x7fc73c1016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:25.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 shutdown_connections 2023-08-25T02:18:25.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:25.203+0000 7fc7432b9700 1 -- 172.21.15.89:0/2363091907 wait complete. 2023-08-25T02:18:25.657 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (2/15)... 2023-08-25T02:18:25.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:25.491+0000 7fb1a9dee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:18:25.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:25.717+0000 7fb1a9dee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:18:26.126 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:26 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2363091907' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-08-25T02:18:26.127 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:25.864+0000 7fb1a9dee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:18:26.581 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:26.125+0000 7fb1a9dee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:18:27.079 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:26.816+0000 7fb1a9dee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:18:27.080 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:26.945+0000 7fb1a9dee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:18:27.469 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:27.136+0000 7fb1a9dee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:18:27.469 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:27.249+0000 7fb1a9dee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:18:28.081 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:27.464+0000 7fb1a9dee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:18:28.426 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:28.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:28.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:18:28.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2023-08-25T02:18:28.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-08-25T02:18:28.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-08-25T02:18:28.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-08-25T02:18:28.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-08-25T02:18:28.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-08-25T02:18:28.429 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-08-25T02:18:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-08-25T02:18:28.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-08-25T02:18:28.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-08-25T02:18:28.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-08-25T02:18:28.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-08-25T02:18:28.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-08-25T02:18:28.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-08-25T02:18:28.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:28.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2023-08-25T02:18:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2023-08-25T02:18:08.578980+0000", 2023-08-25T02:18:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:28.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:28.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-08-25T02:18:28.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:28.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e5705f700 1 Processor -- start 2023-08-25T02:18:28.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e5705f700 1 -- start start 2023-08-25T02:18:28.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e5705f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:28.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e5705f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e50106cd0 con 0x7f9e501062e0 2023-08-25T02:18:28.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e54dfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:28.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e54dfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45366/0 (socket says 172.21.15.89:45366) 2023-08-25T02:18:28.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.029+0000 7f9e54dfb700 1 -- 172.21.15.89:0/3393467210 learned_addr learned my addr 172.21.15.89:0/3393467210 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:28.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.030+0000 7f9e54dfb700 1 -- 172.21.15.89:0/3393467210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e501074e0 con 0x7f9e501062e0 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.030+0000 7f9e54dfb700 1 --2- 172.21.15.89:0/3393467210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e440098a0 tx=0x7f9e44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ac46ccba09df06 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.030+0000 7f9e4f7fe700 1 -- 172.21.15.89:0/3393467210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e44004160 con 0x7f9e501062e0 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.030+0000 7f9e4f7fe700 1 -- 172.21.15.89:0/3393467210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e440042c0 con 0x7f9e501062e0 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.030+0000 7f9e4f7fe700 1 -- 172.21.15.89:0/3393467210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e44011a00 con 0x7f9e501062e0 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 -- 172.21.15.89:0/3393467210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 msgr2=0x7f9e50106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:28.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 --2- 172.21.15.89:0/3393467210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e440098a0 tx=0x7f9e44009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 -- 172.21.15.89:0/3393467210 shutdown_connections 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 --2- 172.21.15.89:0/3393467210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50106700 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 -- 172.21.15.89:0/3393467210 >> 172.21.15.89:0/3393467210 conn(0x7f9e50101860 msgr2=0x7f9e50103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 -- 172.21.15.89:0/3393467210 shutdown_connections 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.031+0000 7f9e5705f700 1 -- 172.21.15.89:0/3393467210 wait complete. 2023-08-25T02:18:28.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e5705f700 1 Processor -- start 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e5705f700 1 -- start start 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e5705f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e5705f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5019a4d0 con 0x7f9e501062e0 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e54dfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e54dfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45368/0 (socket says 172.21.15.89:45368) 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.032+0000 7f9e54dfb700 1 -- 172.21.15.89:0/1565457487 learned_addr learned my addr 172.21.15.89:0/1565457487 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e54dfb700 1 -- 172.21.15.89:0/1565457487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44009550 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e54dfb700 1 --2- 172.21.15.89:0/1565457487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009fb0 tx=0x7f9e44011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e44003930 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e44003a90 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5019a6d0 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e440195e0 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5019aaf0 con 0x7f9e501062e0 2023-08-25T02:18:28.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9e44019b50 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.033+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9e440243f0 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.034+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e50061f50 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.036+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f9e4402de30 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.107+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9e5010a6e0 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.107+0000 7f9e4dffb700 1 -- 172.21.15.89:0/1565457487 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f9e44003c00 con 0x7f9e501062e0 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.108+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 msgr2=0x7f9e50199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.108+0000 7f9e5705f700 1 --2- 172.21.15.89:0/1565457487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009fb0 tx=0x7f9e44011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:28.475 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.108+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 shutdown_connections 2023-08-25T02:18:28.475 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.108+0000 7f9e5705f700 1 --2- 172.21.15.89:0/1565457487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e501062e0 0x7f9e50199f90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:28.480 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.108+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 >> 172.21.15.89:0/1565457487 conn(0x7f9e50101860 msgr2=0x7f9e50102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:28.480 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.109+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 shutdown_connections 2023-08-25T02:18:28.480 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:28.109+0000 7f9e5705f700 1 -- 172.21.15.89:0/1565457487 wait complete. 2023-08-25T02:18:28.481 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (3/15)... 2023-08-25T02:18:28.481 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:28 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1565457487' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-08-25T02:18:28.482 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:28.359+0000 7fb1a9dee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:18:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: Activating manager daemon x 2023-08-25T02:18:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: mgrmap e2: x(active, starting, since 0.00312738s) 2023-08-25T02:18:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:18:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:18:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: Manager daemon x is now available 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' 2023-08-25T02:18:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' 2023-08-25T02:18:29.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:29 smithi089 ceph-mon[98732]: from='mgr.14100 172.21.15.89:0/9248451' entity='mgr.x' 2023-08-25T02:18:30.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:30 smithi089 ceph-mon[98732]: mgrmap e3: x(active, since 1.00921s) 2023-08-25T02:18:31.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:31 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2794839449' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-08-25T02:18:31.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:31.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:31.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:18:31.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2023-08-25T02:18:31.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:31.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-08-25T02:18:31.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2023-08-25T02:18:31.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:31.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-08-25T02:18:31.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-08-25T02:18:31.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:31.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-08-25T02:18:31.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-08-25T02:18:31.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-08-25T02:18:31.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-08-25T02:18:31.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-08-25T02:18:31.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-08-25T02:18:31.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:31.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-08-25T02:18:31.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-08-25T02:18:31.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2023-08-25T02:18:31.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-08-25T02:18:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2023-08-25T02:18:08.578980+0000", 2023-08-25T02:18:31.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2023-08-25T02:18:31.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2023-08-25T02:18:31.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-08-25T02:18:31.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:31.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.855+0000 7f6066d0e700 1 Processor -- start 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.855+0000 7f6066d0e700 1 -- start start 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.855+0000 7f6066d0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.855+0000 7f6066d0e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6060106cd0 con 0x7f60601062e0 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f6064aaa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f6064aaa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45394/0 (socket says 172.21.15.89:45394) 2023-08-25T02:18:31.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f6064aaa700 1 -- 172.21.15.89:0/2280769533 learned_addr learned my addr 172.21.15.89:0/2280769533 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:31.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f6064aaa700 1 -- 172.21.15.89:0/2280769533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60601074e0 con 0x7f60601062e0 2023-08-25T02:18:31.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f6064aaa700 1 --2- 172.21.15.89:0/2280769533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f60540098a0 tx=0x7f6054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20797da68e6b2a3d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:31.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f605f7fe700 1 -- 172.21.15.89:0/2280769533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6054004160 con 0x7f60601062e0 2023-08-25T02:18:31.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f605f7fe700 1 -- 172.21.15.89:0/2280769533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60540042c0 con 0x7f60601062e0 2023-08-25T02:18:31.525 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.856+0000 7f605f7fe700 1 -- 172.21.15.89:0/2280769533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6054011a00 con 0x7f60601062e0 2023-08-25T02:18:31.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- 172.21.15.89:0/2280769533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 msgr2=0x7f6060106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:31.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2280769533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f60540098a0 tx=0x7f6054009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:31.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- 172.21.15.89:0/2280769533 shutdown_connections 2023-08-25T02:18:31.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2280769533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060106700 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:31.526 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- 172.21.15.89:0/2280769533 >> 172.21.15.89:0/2280769533 conn(0x7f6060101860 msgr2=0x7f6060103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- 172.21.15.89:0/2280769533 shutdown_connections 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- 172.21.15.89:0/2280769533 wait complete. 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 Processor -- start 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- start start 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:31.527 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.857+0000 7f6066d0e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606019a4d0 con 0x7f60601062e0 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6064aaa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6064aaa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45396/0 (socket says 172.21.15.89:45396) 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6064aaa700 1 -- 172.21.15.89:0/2794839449 learned_addr learned my addr 172.21.15.89:0/2794839449 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6064aaa700 1 -- 172.21.15.89:0/2794839449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054009550 con 0x7f60601062e0 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6064aaa700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6054009fb0 tx=0x7f6054011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:31.528 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6054003930 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6054003a90 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606019a6d0 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60540195e0 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.858+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606019aaf0 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.859+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 3) v1 ==== 45018+0+0 (secure 0 0 0) 0x7f60540198a0 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.859+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6060061f50 con 0x7f60601062e0 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.859+0000 7f605dffb700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 0x7f605003a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:31.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.859+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f605404bac0 con 0x7f60601062e0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.859+0000 7f605ffff700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 0x7f605003a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.860+0000 7f605ffff700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 0x7f605003a410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:30.864+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f605402b340 con 0x7f60601062e0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.123+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f606019d870 con 0x7f60601062e0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.124+0000 7f605dffb700 1 -- 172.21.15.89:0/2794839449 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6054018b40 con 0x7f60601062e0 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 msgr2=0x7f605003a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:31.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 0x7f605003a410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:31.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 msgr2=0x7f6060199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:31.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6054009fb0 tx=0x7f6054011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:31.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 shutdown_connections 2023-08-25T02:18:31.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f6050037f50 0x7f605003a410 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:31.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 --2- 172.21.15.89:0/2794839449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60601062e0 0x7f6060199f90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:31.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 >> 172.21.15.89:0/2794839449 conn(0x7f6060101860 msgr2=0x7f6060102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:31.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 shutdown_connections 2023-08-25T02:18:31.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.126+0000 7f6066d0e700 1 -- 172.21.15.89:0/2794839449 wait complete. 2023-08-25T02:18:31.533 INFO:teuthology.orchestra.run.smithi089.stdout:mgr is available 2023-08-25T02:18:32.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:32.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2023-08-25T02:18:32.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:32.427 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-08-25T02:18:32.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-08-25T02:18:32.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-08-25T02:18:32.428 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-08-25T02:18:32.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:32.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2023-08-25T02:18:32.430 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-08-25T02:18:32.431 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-08-25T02:18:32.432 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-08-25T02:18:32.432 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6e6737700 1 Processor -- start 2023-08-25T02:18:32.432 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6e6737700 1 -- start start 2023-08-25T02:18:32.432 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6e6737700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:32.437 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6e6737700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0104fe0 con 0x7ff6e01045f0 2023-08-25T02:18:32.437 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:32.438 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45400/0 (socket says 172.21.15.89:45400) 2023-08-25T02:18:32.438 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.917+0000 7ff6dffff700 1 -- 172.21.15.89:0/4116308861 learned_addr learned my addr 172.21.15.89:0/4116308861 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:32.438 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6dffff700 1 -- 172.21.15.89:0/4116308861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0105120 con 0x7ff6e01045f0 2023-08-25T02:18:32.439 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6dffff700 1 --2- 172.21.15.89:0/4116308861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098a0 tx=0x7ff6d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabba76e80efc4c4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:32.439 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6deffd700 1 -- 172.21.15.89:0/4116308861 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6d0004160 con 0x7ff6e01045f0 2023-08-25T02:18:32.439 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6deffd700 1 -- 172.21.15.89:0/4116308861 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6d00042c0 con 0x7ff6e01045f0 2023-08-25T02:18:32.439 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6deffd700 1 -- 172.21.15.89:0/4116308861 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6d0011a00 con 0x7ff6e01045f0 2023-08-25T02:18:32.439 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6e6737700 1 -- 172.21.15.89:0/4116308861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 msgr2=0x7ff6e0104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.918+0000 7ff6e6737700 1 --2- 172.21.15.89:0/4116308861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098a0 tx=0x7ff6d0009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 -- 172.21.15.89:0/4116308861 shutdown_connections 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 --2- 172.21.15.89:0/4116308861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e0104a10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 -- 172.21.15.89:0/4116308861 >> 172.21.15.89:0/4116308861 conn(0x7ff6e00fff00 msgr2=0x7ff6e0102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 -- 172.21.15.89:0/4116308861 shutdown_connections 2023-08-25T02:18:32.440 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 -- 172.21.15.89:0/4116308861 wait complete. 2023-08-25T02:18:32.441 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 Processor -- start 2023-08-25T02:18:32.441 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 -- start start 2023-08-25T02:18:32.441 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.919+0000 7ff6e6737700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:32.441 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6e6737700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e019a540 con 0x7ff6e01045f0 2023-08-25T02:18:32.442 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:32.442 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45402/0 (socket says 172.21.15.89:45402) 2023-08-25T02:18:32.443 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dffff700 1 -- 172.21.15.89:0/3461531388 learned_addr learned my addr 172.21.15.89:0/3461531388 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:32.443 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dffff700 1 -- 172.21.15.89:0/3461531388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009550 con 0x7ff6e01045f0 2023-08-25T02:18:32.443 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dffff700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009fb0 tx=0x7ff6d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:32.443 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6d0003a40 con 0x7ff6e01045f0 2023-08-25T02:18:32.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6d0003ba0 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e019a740 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6d001a750 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.920+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e019ab60 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.921+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 45124+0+0 (secure 0 0 0) 0x7ff6d0024b60 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.921+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e0061f50 con 0x7ff6e01045f0 2023-08-25T02:18:32.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.921+0000 7ff6dd7fa700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 0x7ff6c803a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.921+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff6d004bb80 con 0x7ff6e01045f0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.922+0000 7ff6df7fe700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 0x7ff6c803a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.922+0000 7ff6df7fe700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 0x7ff6c803a530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006fd0 tx=0x7ff6d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:31.926+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff6d0018350 con 0x7ff6e01045f0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.112+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff6e01083b0 con 0x7ff6e01045f0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.113+0000 7ff6dd7fa700 1 -- 172.21.15.89:0/3461531388 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7ff6d004be60 con 0x7ff6e01045f0 2023-08-25T02:18:32.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.115+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 msgr2=0x7ff6c803a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.115+0000 7ff6e6737700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 0x7ff6c803a530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006fd0 tx=0x7ff6d4006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.115+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 msgr2=0x7ff6e019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.115+0000 7ff6e6737700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009fb0 tx=0x7ff6d0011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 shutdown_connections 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff6c8038070 0x7ff6c803a530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 --2- 172.21.15.89:0/3461531388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6e01045f0 0x7ff6e019a000 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 >> 172.21.15.89:0/3461531388 conn(0x7ff6e00fff00 msgr2=0x7ff6e0101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:32.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 shutdown_connections 2023-08-25T02:18:32.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.116+0000 7ff6e6737700 1 -- 172.21.15.89:0/3461531388 wait complete. 2023-08-25T02:18:32.457 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling cephadm module... 2023-08-25T02:18:32.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:32 smithi089 ceph-mon[98732]: mgrmap e4: x(active, since 3s) 2023-08-25T02:18:32.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:32 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3461531388' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-08-25T02:18:33.722 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/640947624' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-08-25T02:18:33.722 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setuser ceph since I am not root 2023-08-25T02:18:33.723 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setgroup ceph since I am not root 2023-08-25T02:18:33.723 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:33.551+0000 7f4d91c16200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:18:33.723 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:33.632+0000 7f4d91c16200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:18:33.724 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:33.721+0000 7f4d91c16200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:18:33.785 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.851+0000 7ff4f57bc700 1 Processor -- start 2023-08-25T02:18:33.785 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.851+0000 7ff4f57bc700 1 -- start start 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.851+0000 7ff4f57bc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.851+0000 7ff4f57bc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0105390 con 0x7ff4f01049a0 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45406/0 (socket says 172.21.15.89:45406) 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4eeffd700 1 -- 172.21.15.89:0/142391958 learned_addr learned my addr 172.21.15.89:0/142391958 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4eeffd700 1 -- 172.21.15.89:0/142391958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f01054d0 con 0x7ff4f01049a0 2023-08-25T02:18:33.786 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4eeffd700 1 --2- 172.21.15.89:0/142391958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098a0 tx=0x7ff4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbbc1394c27ffed server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4edffb700 1 -- 172.21.15.89:0/142391958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4004160 con 0x7ff4f01049a0 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4edffb700 1 -- 172.21.15.89:0/142391958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e40042c0 con 0x7ff4f01049a0 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.852+0000 7ff4edffb700 1 -- 172.21.15.89:0/142391958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4011a00 con 0x7ff4f01049a0 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 -- 172.21.15.89:0/142391958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 msgr2=0x7ff4f0104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/142391958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098a0 tx=0x7ff4e4009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 -- 172.21.15.89:0/142391958 shutdown_connections 2023-08-25T02:18:33.787 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/142391958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f0104dc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:33.788 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 -- 172.21.15.89:0/142391958 >> 172.21.15.89:0/142391958 conn(0x7ff4f00ffae0 msgr2=0x7ff4f0101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:33.788 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 -- 172.21.15.89:0/142391958 shutdown_connections 2023-08-25T02:18:33.788 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 -- 172.21.15.89:0/142391958 wait complete. 2023-08-25T02:18:33.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.853+0000 7ff4f57bc700 1 Processor -- start 2023-08-25T02:18:33.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4f57bc700 1 -- start start 2023-08-25T02:18:33.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4f57bc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:33.798 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4f57bc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f0190030 con 0x7ff4f01049a0 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4eeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45408/0 (socket says 172.21.15.89:45408) 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4eeffd700 1 -- 172.21.15.89:0/640947624 learned_addr learned my addr 172.21.15.89:0/640947624 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4eeffd700 1 -- 172.21.15.89:0/640947624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4009550 con 0x7ff4f01049a0 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4eeffd700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fb0 tx=0x7ff4e4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.854+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4003a40 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e4003ba0 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f018e140 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e401a750 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f018e560 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 45124+0+0 (secure 0 0 0) 0x7ff4e401a980 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.855+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f0061f50 con 0x7ff4f01049a0 2023-08-25T02:18:33.800 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.856+0000 7ff4d7fff700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 0x7ff4d803a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.856+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff4e404bb70 con 0x7ff4f01049a0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.856+0000 7ff4ee7fc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 0x7ff4d803a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.856+0000 7ff4ee7fc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 0x7ff4d803a580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006fd0 tx=0x7ff4e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:32.860+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff4e401ac30 con 0x7ff4f01049a0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.099+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff4f0108de0 con 0x7ff4f01049a0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.389+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 5) v1 ==== 45135+0+0 (secure 0 0 0) 0x7ff4e402d910 con 0x7ff4f01049a0 2023-08-25T02:18:33.801 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.390+0000 7ff4d7fff700 1 -- 172.21.15.89:0/640947624 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff4e404b8d0 con 0x7ff4f01049a0 2023-08-25T02:18:33.802 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 msgr2=0x7ff4d803a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:33.802 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 0x7ff4d803a580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006fd0 tx=0x7ff4e0006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:33.802 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 msgr2=0x7ff4f018faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:33.802 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fb0 tx=0x7ff4e4011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:33.802 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 shutdown_connections 2023-08-25T02:18:33.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7ff4d80380c0 0x7ff4d803a580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:33.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 --2- 172.21.15.89:0/640947624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f01049a0 0x7ff4f018faf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:33.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.392+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 >> 172.21.15.89:0/640947624 conn(0x7ff4f00ffae0 msgr2=0x7ff4f01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:33.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.393+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 shutdown_connections 2023-08-25T02:18:33.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:33.393+0000 7ff4f57bc700 1 -- 172.21.15.89:0/640947624 wait complete. 2023-08-25T02:18:33.994 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:33.822+0000 7f4d91c16200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:18:34.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:33.997+0000 7f4d91c16200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:18:34.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/640947624' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-08-25T02:18:34.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-mon[98732]: mgrmap e5: x(active, since 5s) 2023-08-25T02:18:34.669 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:34.383+0000 7f4d91c16200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:18:34.670 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:34.519+0000 7f4d91c16200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:18:34.670 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:34.614+0000 7f4d91c16200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:18:34.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:34.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-08-25T02:18:34.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb4bd7a700 1 Processor -- start 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb4bd7a700 1 -- start start 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb4bd7a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:34.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb4bd7a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440740e0 con 0x7feb44104bd0 2023-08-25T02:18:34.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb49b16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:34.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb49b16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45422/0 (socket says 172.21.15.89:45422) 2023-08-25T02:18:34.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.280+0000 7feb49b16700 1 -- 172.21.15.89:0/3336293932 learned_addr learned my addr 172.21.15.89:0/3336293932 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:34.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.281+0000 7feb49b16700 1 -- 172.21.15.89:0/3336293932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44105530 con 0x7feb44104bd0 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.281+0000 7feb49b16700 1 --2- 172.21.15.89:0/3336293932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ab1395647a351a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.281+0000 7feb48b14700 1 -- 172.21.15.89:0/3336293932 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb40004160 con 0x7feb44104bd0 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.281+0000 7feb48b14700 1 -- 172.21.15.89:0/3336293932 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb400042c0 con 0x7feb44104bd0 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.281+0000 7feb48b14700 1 -- 172.21.15.89:0/3336293932 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb40011a00 con 0x7feb44104bd0 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- 172.21.15.89:0/3336293932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 msgr2=0x7feb44104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:34.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/3336293932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7feb400098a0 tx=0x7feb40009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- 172.21.15.89:0/3336293932 shutdown_connections 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/3336293932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb44104ff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- 172.21.15.89:0/3336293932 >> 172.21.15.89:0/3336293932 conn(0x7feb44100770 msgr2=0x7feb44102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- 172.21.15.89:0/3336293932 shutdown_connections 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- 172.21.15.89:0/3336293932 wait complete. 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 Processor -- start 2023-08-25T02:18:34.930 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- start start 2023-08-25T02:18:34.931 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:34.931 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.282+0000 7feb4bd7a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4419c710 con 0x7feb44104bd0 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb49b16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb49b16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45424/0 (socket says 172.21.15.89:45424) 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb49b16700 1 -- 172.21.15.89:0/2489745975 learned_addr learned my addr 172.21.15.89:0/2489745975 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb49b16700 1 -- 172.21.15.89:0/2489745975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009550 con 0x7feb44104bd0 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb49b16700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb40009fb0 tx=0x7feb40011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:34.932 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb40003a40 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb40003ba0 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb4001a750 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4419c910 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.283+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4419cd30 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 45135+0+0 (secure 0 0 0) 0x7feb40024b60 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44195a50 con 0x7feb44104bd0 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb3affd700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 0x7feb3003a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:34.933 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb49315700 1 -- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 msgr2=0x7feb3003a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb4004bbd0 con 0x7feb44104bd0 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.284+0000 7feb49315700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 0x7feb3003a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.289+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7feb40024430 con 0x7feb44104bd0 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.485+0000 7feb49315700 1 -- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 msgr2=0x7feb3003a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.485+0000 7feb49315700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 0x7feb3003a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.538+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7feb44061fd0 con 0x7feb44104bd0 2023-08-25T02:18:34.934 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.538+0000 7feb3affd700 1 -- 172.21.15.89:0/2489745975 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7feb40024430 con 0x7feb44104bd0 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 msgr2=0x7feb3003a5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 0x7feb3003a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 msgr2=0x7feb4419c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb40009fb0 tx=0x7feb40011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 shutdown_connections 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7feb30038110 0x7feb3003a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 --2- 172.21.15.89:0/2489745975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb44104bd0 0x7feb4419c1d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:34.935 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 >> 172.21.15.89:0/2489745975 conn(0x7feb44100770 msgr2=0x7feb44190790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:34.936 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.540+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 shutdown_connections 2023-08-25T02:18:34.937 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:34.541+0000 7feb4bd7a700 1 -- 172.21.15.89:0/2489745975 wait complete. 2023-08-25T02:18:34.937 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2023-08-25T02:18:34.937 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 5... 2023-08-25T02:18:35.079 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:34 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:34.713+0000 7f4d91c16200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:18:35.392 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:35 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:35.149+0000 7f4d91c16200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:18:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:35 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2489745975' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-08-25T02:18:36.080 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:35 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:35.811+0000 7f4d91c16200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:18:36.080 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:35 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:35.893+0000 7f4d91c16200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:18:36.918 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:36 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:36.654+0000 7f4d91c16200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:18:36.919 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:36 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:36.744+0000 7f4d91c16200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:18:37.195 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:36 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:36.916+0000 7f4d91c16200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:18:37.564 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:37 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:37.192+0000 7f4d91c16200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:18:37.565 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:37 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:37.416+0000 7f4d91c16200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:18:37.565 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:37 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:37.562+0000 7f4d91c16200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:18:38.080 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:37 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:37.822+0000 7f4d91c16200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:18:38.835 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:38.513+0000 7f4d91c16200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:18:38.835 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:38.641+0000 7f4d91c16200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:18:39.157 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:38.831+0000 7f4d91c16200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:18:39.157 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:38.942+0000 7f4d91c16200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:18:40.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:39 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:39.155+0000 7f4d91c16200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:18:40.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:40 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:40.003+0000 7f4d91c16200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:18:41.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: Active manager daemon x restarted 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: Activating manager daemon x 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: osdmap e2: 0 total, 0 up, 0 in 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: mgrmap e6: x(active, starting, since 0.0032472s) 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:18:41.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: Manager daemon x is now available 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: Found migration_current of "None". Setting to last migration. 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:41.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:41.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:41.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:18:41.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:18:41.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:41.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:41 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:41.387 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-08-25T02:18:41.389 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2023-08-25T02:18:41.389 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:41.390 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.381+0000 7f31a68f6700 1 Processor -- start 2023-08-25T02:18:41.390 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.381+0000 7f31a68f6700 1 -- start start 2023-08-25T02:18:41.391 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f31a68f6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:41.393 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f31a68f6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0074120 con 0x7f31a0106da0 2023-08-25T02:18:41.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:41.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45432/0 (socket says 172.21.15.89:45432) 2023-08-25T02:18:41.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319ffff700 1 -- 172.21.15.89:0/1923787333 learned_addr learned my addr 172.21.15.89:0/1923787333 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:41.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319ffff700 1 -- 172.21.15.89:0/1923787333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a01096d0 con 0x7f31a0106da0 2023-08-25T02:18:41.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319ffff700 1 --2- 172.21.15.89:0/1923787333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f3190009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f18a5c53a11263 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:41.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319effd700 1 -- 172.21.15.89:0/1923787333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3190004160 con 0x7f31a0106da0 2023-08-25T02:18:41.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319effd700 1 -- 172.21.15.89:0/1923787333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31900042c0 con 0x7f31a0106da0 2023-08-25T02:18:41.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.382+0000 7f319effd700 1 -- 172.21.15.89:0/1923787333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3190011a00 con 0x7f31a0106da0 2023-08-25T02:18:41.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 -- 172.21.15.89:0/1923787333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 msgr2=0x7f31a0109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 --2- 172.21.15.89:0/1923787333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f3190009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 -- 172.21.15.89:0/1923787333 shutdown_connections 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 --2- 172.21.15.89:0/1923787333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0109190 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 -- 172.21.15.89:0/1923787333 >> 172.21.15.89:0/1923787333 conn(0x7f31a0100770 msgr2=0x7f31a0102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 -- 172.21.15.89:0/1923787333 shutdown_connections 2023-08-25T02:18:41.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.383+0000 7f31a68f6700 1 -- 172.21.15.89:0/1923787333 wait complete. 2023-08-25T02:18:41.400 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.384+0000 7f31a68f6700 1 Processor -- start 2023-08-25T02:18:41.400 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.384+0000 7f31a68f6700 1 -- start start 2023-08-25T02:18:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f31a68f6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f31a68f6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019a4e0 con 0x7f31a0106da0 2023-08-25T02:18:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45434/0 (socket says 172.21.15.89:45434) 2023-08-25T02:18:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319ffff700 1 -- 172.21.15.89:0/2308497755 learned_addr learned my addr 172.21.15.89:0/2308497755 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319ffff700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190009550 con 0x7f31a0106da0 2023-08-25T02:18:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319ffff700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3190009130 tx=0x7f3190011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3190003b70 con 0x7f31a0106da0 2023-08-25T02:18:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a019a6e0 con 0x7f31a0106da0 2023-08-25T02:18:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3190003cd0 con 0x7f31a0106da0 2023-08-25T02:18:41.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31900197d0 con 0x7f31a0106da0 2023-08-25T02:18:41.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.385+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a019ab00 con 0x7f31a0106da0 2023-08-25T02:18:41.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.386+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f3190019a10 con 0x7f31a0106da0 2023-08-25T02:18:41.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.386+0000 7f319d7fa700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:41.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.386+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318803ace0 con 0x7f3188038110 2023-08-25T02:18:41.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.387+0000 7f319f7fe700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:41.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.387+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f319004c0f0 con 0x7f31a0106da0 2023-08-25T02:18:41.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.387+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:18:41.413 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.588+0000 7f319f7fe700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:41.413 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.588+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:18:41.414 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.989+0000 7f319f7fe700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:41.419 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:35.989+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T02:18:41.420 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:36.790+0000 7f319f7fe700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:41.420 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:36.790+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T02:18:41.420 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:38.391+0000 7f319f7fe700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/730683542 2023-08-25T02:18:41.420 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:38.391+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T02:18:41.421 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:40.019+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 6) v1 ==== 44902+0+0 (secure 0 0 0) 0x7f319000b900 con 0x7f31a0106da0 2023-08-25T02:18:41.421 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:40.019+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 msgr2=0x7f318803a5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:18:41.422 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:40.019+0000 7f319d7fa700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.422 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.020+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 7) v1 ==== 45029+0+0 (secure 0 0 0) 0x7f3190019cc0 con 0x7f31a0106da0 2023-08-25T02:18:41.422 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.020+0000 7f319d7fa700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 0x7f318803de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:41.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.020+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318803ace0 con 0x7f318803ba30 2023-08-25T02:18:41.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.020+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 0x7f318803de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:41.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.021+0000 7f319f7fe700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 0x7f318803de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3194003a10 tx=0x7f3194009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:41.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.022+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f318803ace0 con 0x7f318803ba30 2023-08-25T02:18:41.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.032+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f31a019b2e0 con 0x7f318803ba30 2023-08-25T02:18:41.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.032+0000 7f319d7fa700 1 -- 172.21.15.89:0/2308497755 <== mgr.14120 v2:172.21.15.89:6800/3825317465 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f31a019b2e0 con 0x7f318803ba30 2023-08-25T02:18:41.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 msgr2=0x7f318803de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 0x7f318803de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3194003a10 tx=0x7f3194009290 comp rx=0 tx=0).stop 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 msgr2=0x7f31a0199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3190009130 tx=0x7f3190011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 shutdown_connections 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f318803ba30 0x7f318803de20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a0106da0 0x7f31a0199fa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 --2- 172.21.15.89:0/2308497755 >> [v2:172.21.15.89:6800/730683542,v1:172.21.15.89:6801/730683542] conn(0x7f3188038110 0x7f318803a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:41.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 >> 172.21.15.89:0/2308497755 conn(0x7f31a0100770 msgr2=0x7f31a0101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:41.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 shutdown_connections 2023-08-25T02:18:41.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.033+0000 7f31a68f6700 1 -- 172.21.15.89:0/2308497755 wait complete. 2023-08-25T02:18:41.458 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 5 is available 2023-08-25T02:18:41.458 INFO:teuthology.orchestra.run.smithi089.stdout:Setting orchestrator backend to cephadm... 2023-08-25T02:18:42.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:42 smithi089 ceph-mon[98732]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-08-25T02:18:42.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:42 smithi089 ceph-mon[98732]: mgrmap e7: x(active, since 1.00523s) 2023-08-25T02:18:42.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:42 smithi089 ceph-mon[98732]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-08-25T02:18:42.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:42 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:42.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.878+0000 7f7ad59ee700 1 Processor -- start 2023-08-25T02:18:42.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7ad59ee700 1 -- start start 2023-08-25T02:18:42.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7ad59ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:42.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7ad59ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01071d0 con 0x7f7ad01067e0 2023-08-25T02:18:42.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7aceffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:42.529 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7aceffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45486/0 (socket says 172.21.15.89:45486) 2023-08-25T02:18:42.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7aceffd700 1 -- 172.21.15.89:0/3220391033 learned_addr learned my addr 172.21.15.89:0/3220391033 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:42.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7aceffd700 1 -- 172.21.15.89:0/3220391033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad01053f0 con 0x7f7ad01067e0 2023-08-25T02:18:42.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7aceffd700 1 --2- 172.21.15.89:0/3220391033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00098a0 tx=0x7f7ac0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdb79167c6a417a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:42.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.879+0000 7f7acdffb700 1 -- 172.21.15.89:0/3220391033 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ac0004160 con 0x7f7ad01067e0 2023-08-25T02:18:42.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7acdffb700 1 -- 172.21.15.89:0/3220391033 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ac00042c0 con 0x7f7ad01067e0 2023-08-25T02:18:42.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7acdffb700 1 -- 172.21.15.89:0/3220391033 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ac0011a00 con 0x7f7ad01067e0 2023-08-25T02:18:42.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3220391033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 msgr2=0x7f7ad0106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:42.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3220391033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00098a0 tx=0x7f7ac0009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:42.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3220391033 shutdown_connections 2023-08-25T02:18:42.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3220391033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0106c00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:42.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3220391033 >> 172.21.15.89:0/3220391033 conn(0x7f7ad00fff00 msgr2=0x7f7ad0102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:42.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3220391033 shutdown_connections 2023-08-25T02:18:42.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.880+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3220391033 wait complete. 2023-08-25T02:18:42.536 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7ad59ee700 1 Processor -- start 2023-08-25T02:18:42.536 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7ad59ee700 1 -- start start 2023-08-25T02:18:42.537 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7ad59ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:42.537 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7ad59ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0199410 con 0x7f7ad01067e0 2023-08-25T02:18:42.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7aceffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:42.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7aceffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45488/0 (socket says 172.21.15.89:45488) 2023-08-25T02:18:42.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.881+0000 7f7aceffd700 1 -- 172.21.15.89:0/3805990303 learned_addr learned my addr 172.21.15.89:0/3805990303 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:42.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7aceffd700 1 -- 172.21.15.89:0/3805990303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0009550 con 0x7f7ad01067e0 2023-08-25T02:18:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7aceffd700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009130 tx=0x7f7ac0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ac0003ab0 con 0x7f7ad01067e0 2023-08-25T02:18:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ac0003c10 con 0x7f7ad01067e0 2023-08-25T02:18:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad0199610 con 0x7f7ad01067e0 2023-08-25T02:18:42.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ac00247d0 con 0x7f7ad01067e0 2023-08-25T02:18:42.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.882+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad0199a30 con 0x7f7ad01067e0 2023-08-25T02:18:42.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.883+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad0061f50 con 0x7f7ad01067e0 2023-08-25T02:18:42.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.883+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 45029+0+0 (secure 0 0 0) 0x7f7ac001abb0 con 0x7f7ad01067e0 2023-08-25T02:18:42.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.883+0000 7f7ad49ec700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 0x7f7abc03a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:42.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.883+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7ac004be30 con 0x7f7ad01067e0 2023-08-25T02:18:42.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.883+0000 7f7ace7fc700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 0x7f7abc03a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:42.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.884+0000 7f7ace7fc700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 0x7f7abc03a4f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8006fd0 tx=0x7f7ab8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:42.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:41.888+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7ac0018920 con 0x7f7ad01067e0 2023-08-25T02:18:42.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.090+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ad0102290 con 0x7f7abc038030 2023-08-25T02:18:42.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.096+0000 7f7ad49ec700 1 -- 172.21.15.89:0/3805990303 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7ad0102290 con 0x7f7abc038030 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 msgr2=0x7f7abc03a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 0x7f7abc03a4f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8006fd0 tx=0x7f7ab8006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 msgr2=0x7f7ad0198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009130 tx=0x7f7ac0011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 shutdown_connections 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7abc038030 0x7f7abc03a4f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:42.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 --2- 172.21.15.89:0/3805990303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad01067e0 0x7f7ad0198ed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:42.546 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 >> 172.21.15.89:0/3805990303 conn(0x7f7ad00fff00 msgr2=0x7f7ad0101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:42.546 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 shutdown_connections 2023-08-25T02:18:42.546 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.099+0000 7f7ad59ee700 1 -- 172.21.15.89:0/3805990303 wait complete. 2023-08-25T02:18:43.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout value unchanged 2023-08-25T02:18:43.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa095ba5700 1 Processor -- start 2023-08-25T02:18:43.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa095ba5700 1 -- start start 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa095ba5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa095ba5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0901071d0 con 0x7fa0901067e0 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45492/0 (socket says 172.21.15.89:45492) 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa08f7fe700 1 -- 172.21.15.89:0/2821167578 learned_addr learned my addr 172.21.15.89:0/2821167578 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa08f7fe700 1 -- 172.21.15.89:0/2821167578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0901053f0 con 0x7fa0901067e0 2023-08-25T02:18:43.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.954+0000 7fa08f7fe700 1 --2- 172.21.15.89:0/2821167578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0800098a0 tx=0x7fa080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220f62ffb81c0297 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa08e7fc700 1 -- 172.21.15.89:0/2821167578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa080004160 con 0x7fa0901067e0 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa08e7fc700 1 -- 172.21.15.89:0/2821167578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0800042c0 con 0x7fa0901067e0 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa08e7fc700 1 -- 172.21.15.89:0/2821167578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa080011a00 con 0x7fa0901067e0 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 -- 172.21.15.89:0/2821167578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 msgr2=0x7fa090106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2821167578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa0800098a0 tx=0x7fa080009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 -- 172.21.15.89:0/2821167578 shutdown_connections 2023-08-25T02:18:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2821167578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090106c00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 -- 172.21.15.89:0/2821167578 >> 172.21.15.89:0/2821167578 conn(0x7fa0900fff00 msgr2=0x7fa090102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 -- 172.21.15.89:0/2821167578 shutdown_connections 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.955+0000 7fa095ba5700 1 -- 172.21.15.89:0/2821167578 wait complete. 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa095ba5700 1 Processor -- start 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa095ba5700 1 -- start start 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa095ba5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa095ba5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09019cd80 con 0x7fa0901067e0 2023-08-25T02:18:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45494/0 (socket says 172.21.15.89:45494) 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.956+0000 7fa08f7fe700 1 -- 172.21.15.89:0/2544377950 learned_addr learned my addr 172.21.15.89:0/2544377950 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa08f7fe700 1 -- 172.21.15.89:0/2544377950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080009550 con 0x7fa0901067e0 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa08f7fe700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa080009fb0 tx=0x7fa080011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa080003a40 con 0x7fa0901067e0 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09019cec0 con 0x7fa0901067e0 2023-08-25T02:18:43.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa080003ba0 con 0x7fa0901067e0 2023-08-25T02:18:43.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa08001a750 con 0x7fa0901067e0 2023-08-25T02:18:43.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.957+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09019d280 con 0x7fa0901067e0 2023-08-25T02:18:43.468 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 45029+0+0 (secure 0 0 0) 0x7fa080024b60 con 0x7fa0901067e0 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa090061f50 con 0x7fa0901067e0 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa08cff9700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 0x7fa07c03a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa08004bf50 con 0x7fa0901067e0 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa08effd700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 0x7fa07c03a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.958+0000 7fa08effd700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 0x7fa07c03a460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:43.469 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:42.963+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa080018530 con 0x7fa0901067e0 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.022+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7fa08002d720 con 0x7fa0901067e0 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.154+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fa090102260 con 0x7fa07c037fa0 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.154+0000 7fa08cff9700 1 -- 172.21.15.89:0/2544377950 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fa090102260 con 0x7fa07c037fa0 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 msgr2=0x7fa07c03a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 0x7fa07c03a460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 msgr2=0x7fa090104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:43.470 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa080009fb0 tx=0x7fa080011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:43.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 shutdown_connections 2023-08-25T02:18:43.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fa07c037fa0 0x7fa07c03a460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:43.471 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 --2- 172.21.15.89:0/2544377950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0901067e0 0x7fa090104e40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:43.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.156+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 >> 172.21.15.89:0/2544377950 conn(0x7fa0900fff00 msgr2=0x7fa090101640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:43.472 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.157+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 shutdown_connections 2023-08-25T02:18:43.473 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.157+0000 7fa095ba5700 1 -- 172.21.15.89:0/2544377950 wait complete. 2023-08-25T02:18:43.473 INFO:teuthology.orchestra.run.smithi089.stdout:Generating ssh key... 2023-08-25T02:18:43.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: [25/Aug/2023:02:18:41] ENGINE Bus STARTING 2023-08-25T02:18:43.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: [25/Aug/2023:02:18:41] ENGINE Serving on http://172.21.15.89:8765 2023-08-25T02:18:43.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: [25/Aug/2023:02:18:41] ENGINE Serving on https://172.21.15.89:7150 2023-08-25T02:18:43.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: [25/Aug/2023:02:18:41] ENGINE Bus STARTED 2023-08-25T02:18:43.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:43.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:43.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:43.494 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:43 smithi089 ceph-mon[98732]: mgrmap e8: x(active, since 3s) 2023-08-25T02:18:44.506 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-mon[98732]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:44.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: Generating public/private rsa key pair. 2023-08-25T02:18:44.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: Your identification has been saved in /tmp/tmp3z67c3l2/key. 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: Your public key has been saved in /tmp/tmp3z67c3l2/key.pub. 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: The key fingerprint is: 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: SHA256:Bg0sz1IaGA1iQXUMRsY+wE3SLsQgpgCO+eBGsoOGAKc ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: The key's randomart image is: 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: +---[RSA 3072]----+ 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |&*%%oo. | 2023-08-25T02:18:44.830 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |XX=+=.oo | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |Eo+ B. . | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |B= +o o. | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |=+o .. S | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: |o. . | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: | | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: | | 2023-08-25T02:18:44.831 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: | | 2023-08-25T02:18:44.833 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:44 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: +----[SHA256]-----+ 2023-08-25T02:18:44.904 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.884+0000 7f9ef3a2f700 1 Processor -- start 2023-08-25T02:18:44.904 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.884+0000 7f9ef3a2f700 1 -- start start 2023-08-25T02:18:44.904 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.884+0000 7f9ef3a2f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.884+0000 7f9ef3a2f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0740e0 con 0x7f9eec104bd0 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.884+0000 7f9ef17cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ef17cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45498/0 (socket says 172.21.15.89:45498) 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ef17cb700 1 -- 172.21.15.89:0/1202959123 learned_addr learned my addr 172.21.15.89:0/1202959123 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ef17cb700 1 -- 172.21.15.89:0/1202959123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec105530 con 0x7f9eec104bd0 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ef17cb700 1 --2- 172.21.15.89:0/1202959123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0098a0 tx=0x7f9edc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a145b0a3478b2abb server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ee3fff700 1 -- 172.21.15.89:0/1202959123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9edc004160 con 0x7f9eec104bd0 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ee3fff700 1 -- 172.21.15.89:0/1202959123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9edc0042c0 con 0x7f9eec104bd0 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.885+0000 7f9ee3fff700 1 -- 172.21.15.89:0/1202959123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9edc011a00 con 0x7f9eec104bd0 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/1202959123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 msgr2=0x7f9eec104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/1202959123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0098a0 tx=0x7f9edc009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/1202959123 shutdown_connections 2023-08-25T02:18:44.906 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/1202959123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec104ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:44.907 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/1202959123 >> 172.21.15.89:0/1202959123 conn(0x7f9eec100770 msgr2=0x7f9eec102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:44.908 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/1202959123 shutdown_connections 2023-08-25T02:18:44.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/1202959123 wait complete. 2023-08-25T02:18:44.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 Processor -- start 2023-08-25T02:18:44.913 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- start start 2023-08-25T02:18:44.913 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:44.913 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.886+0000 7f9ef3a2f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec19c710 con 0x7f9eec104bd0 2023-08-25T02:18:44.913 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef17cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef17cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45500/0 (socket says 172.21.15.89:45500) 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef17cb700 1 -- 172.21.15.89:0/3028405205 learned_addr learned my addr 172.21.15.89:0/3028405205 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef17cb700 1 -- 172.21.15.89:0/3028405205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc009550 con 0x7f9eec104bd0 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef17cb700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009130 tx=0x7f9edc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9edc003b70 con 0x7f9eec104bd0 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec19c910 con 0x7f9eec104bd0 2023-08-25T02:18:44.914 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9edc003cd0 con 0x7f9eec104bd0 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9edc0197d0 con 0x7f9eec104bd0 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.887+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec19cd30 con 0x7f9eec104bd0 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.888+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f9edc019a10 con 0x7f9eec104bd0 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.888+0000 7f9ee27fc700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 0x7f9ed803a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.888+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9edc04c1a0 con 0x7f9eec104bd0 2023-08-25T02:18:44.915 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.888+0000 7f9ef0fca700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 0x7f9ed803a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:44.918 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.888+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec195a50 con 0x7f9eec104bd0 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.889+0000 7f9ef0fca700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 0x7f9ed803a5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8006fd0 tx=0x7f9ee8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:43.893+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f9edc02a080 con 0x7f9eec104bd0 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.082+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eec060d60 con 0x7f9ed8038110 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.509+0000 7f9ee27fc700 1 -- 172.21.15.89:0/3028405205 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9eec060d60 con 0x7f9ed8038110 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.511+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 msgr2=0x7f9ed803a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.511+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 0x7f9ed803a5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8006fd0 tx=0x7f9ee8006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:44.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.511+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 msgr2=0x7f9eec19c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:44.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.511+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009130 tx=0x7f9edc011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:44.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 shutdown_connections 2023-08-25T02:18:44.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f9ed8038110 0x7f9ed803a5d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:44.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 --2- 172.21.15.89:0/3028405205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eec104bd0 0x7f9eec19c1d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:44.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 >> 172.21.15.89:0/3028405205 conn(0x7f9eec100770 msgr2=0x7f9eec190790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:44.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 shutdown_connections 2023-08-25T02:18:44.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:44.512+0000 7f9ef3a2f700 1 -- 172.21.15.89:0/3028405205 wait complete. 2023-08-25T02:18:45.132 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:45 smithi089 ceph-mon[98732]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:45.132 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:45 smithi089 ceph-mon[98732]: Generating ssh key... 2023-08-25T02:18:45.132 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:45 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:45.133 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:45 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:45.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:18:45.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f76261b9700 1 Processor -- start 2023-08-25T02:18:45.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f76261b9700 1 -- start start 2023-08-25T02:18:45.849 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f76261b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:45.849 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f76261b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7620074120 con 0x7f7620106da0 2023-08-25T02:18:45.850 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f761f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:45.850 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f761f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45504/0 (socket says 172.21.15.89:45504) 2023-08-25T02:18:45.850 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f761f7fe700 1 -- 172.21.15.89:0/4096085750 learned_addr learned my addr 172.21.15.89:0/4096085750 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:45.850 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f761f7fe700 1 -- 172.21.15.89:0/4096085750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76201096d0 con 0x7f7620106da0 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.336+0000 7f761f7fe700 1 --2- 172.21.15.89:0/4096085750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76100098a0 tx=0x7f7610009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6bfd2dd9c70e49 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f761e7fc700 1 -- 172.21.15.89:0/4096085750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7610004160 con 0x7f7620106da0 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f761e7fc700 1 -- 172.21.15.89:0/4096085750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76100042c0 con 0x7f7620106da0 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f761e7fc700 1 -- 172.21.15.89:0/4096085750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7610011a00 con 0x7f7620106da0 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 -- 172.21.15.89:0/4096085750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 msgr2=0x7f7620109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 --2- 172.21.15.89:0/4096085750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76100098a0 tx=0x7f7610009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:45.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 -- 172.21.15.89:0/4096085750 shutdown_connections 2023-08-25T02:18:45.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 --2- 172.21.15.89:0/4096085750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620109190 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:45.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 -- 172.21.15.89:0/4096085750 >> 172.21.15.89:0/4096085750 conn(0x7f7620100770 msgr2=0x7f7620102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:45.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 -- 172.21.15.89:0/4096085750 shutdown_connections 2023-08-25T02:18:45.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.337+0000 7f76261b9700 1 -- 172.21.15.89:0/4096085750 wait complete. 2023-08-25T02:18:45.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f76261b9700 1 Processor -- start 2023-08-25T02:18:45.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f76261b9700 1 -- start start 2023-08-25T02:18:45.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f76261b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:45.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f76261b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762019a4e0 con 0x7f7620106da0 2023-08-25T02:18:45.856 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f761f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:45.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f761f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45506/0 (socket says 172.21.15.89:45506) 2023-08-25T02:18:45.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f761f7fe700 1 -- 172.21.15.89:0/3292424209 learned_addr learned my addr 172.21.15.89:0/3292424209 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:45.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.338+0000 7f761f7fe700 1 -- 172.21.15.89:0/3292424209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610009550 con 0x7f7620106da0 2023-08-25T02:18:45.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.339+0000 7f761f7fe700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7610009fb0 tx=0x7f7610011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:45.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.339+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7610003a40 con 0x7f7620106da0 2023-08-25T02:18:45.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.339+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762019a6e0 con 0x7f7620106da0 2023-08-25T02:18:45.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.339+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762019ab00 con 0x7f7620106da0 2023-08-25T02:18:45.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7610003ba0 con 0x7f7620106da0 2023-08-25T02:18:45.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f761001a780 con 0x7f7620106da0 2023-08-25T02:18:45.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f761001a8e0 con 0x7f7620106da0 2023-08-25T02:18:45.867 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761cff9700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 0x7f760803a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:45.867 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f761004d0f0 con 0x7f7620106da0 2023-08-25T02:18:45.867 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f761effd700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 0x7f760803a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:45.867 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.340+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7620061f50 con 0x7f7620106da0 2023-08-25T02:18:45.867 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.341+0000 7f761effd700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 0x7f760803a670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7614006fd0 tx=0x7f7614006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.345+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7610023390 con 0x7f7620106da0 2023-08-25T02:18:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.532+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7620102aa0 con 0x7f76080381b0 2023-08-25T02:18:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.533+0000 7f761cff9700 1 -- 172.21.15.89:0/3292424209 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7620102aa0 con 0x7f76080381b0 2023-08-25T02:18:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 msgr2=0x7f760803a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 0x7f760803a670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7614006fd0 tx=0x7f7614006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 msgr2=0x7f7620199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7610009fb0 tx=0x7f7610011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 shutdown_connections 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f76080381b0 0x7f760803a670 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 --2- 172.21.15.89:0/3292424209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7620106da0 0x7f7620199fa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 >> 172.21.15.89:0/3292424209 conn(0x7f7620100770 msgr2=0x7f7620101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 shutdown_connections 2023-08-25T02:18:45.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:45.535+0000 7f76261b9700 1 -- 172.21.15.89:0/3292424209 wait complete. 2023-08-25T02:18:45.870 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-08-25T02:18:45.870 INFO:teuthology.orchestra.run.smithi089.stdout:Adding key to root@localhost authorized_keys... 2023-08-25T02:18:45.870 INFO:teuthology.orchestra.run.smithi089.stdout:Adding host smithi089... 2023-08-25T02:18:46.771 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:46 smithi089 ceph-mon[98732]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:47.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:47 smithi089 ceph-mon[98732]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:47.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:47 smithi089 ceph-mon[98732]: Deploying cephadm binary to smithi089 2023-08-25T02:18:49.103 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Added host 'smithi089' with addr '172.21.15.89' 2023-08-25T02:18:49.103 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.304+0000 7f1a59b2a700 1 Processor -- start 2023-08-25T02:18:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a59b2a700 1 -- start start 2023-08-25T02:18:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a59b2a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a59b2a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a541071d0 con 0x7f1a541067e0 2023-08-25T02:18:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45510/0 (socket says 172.21.15.89:45510) 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a537fe700 1 -- 172.21.15.89:0/603448828 learned_addr learned my addr 172.21.15.89:0/603448828 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a537fe700 1 -- 172.21.15.89:0/603448828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a541053f0 con 0x7f1a541067e0 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a537fe700 1 --2- 172.21.15.89:0/603448828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1a480098a0 tx=0x7f1a48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5050f364a41be11 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a527fc700 1 -- 172.21.15.89:0/603448828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a48004160 con 0x7f1a541067e0 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a527fc700 1 -- 172.21.15.89:0/603448828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a480042c0 con 0x7f1a541067e0 2023-08-25T02:18:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.305+0000 7f1a527fc700 1 -- 172.21.15.89:0/603448828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a48011a00 con 0x7f1a541067e0 2023-08-25T02:18:49.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 -- 172.21.15.89:0/603448828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 msgr2=0x7f1a54106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:49.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/603448828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1a480098a0 tx=0x7f1a48009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:49.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 -- 172.21.15.89:0/603448828 shutdown_connections 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/603448828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54106c00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 -- 172.21.15.89:0/603448828 >> 172.21.15.89:0/603448828 conn(0x7f1a540fff00 msgr2=0x7f1a54102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 -- 172.21.15.89:0/603448828 shutdown_connections 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.306+0000 7f1a59b2a700 1 -- 172.21.15.89:0/603448828 wait complete. 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a59b2a700 1 Processor -- start 2023-08-25T02:18:49.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a59b2a700 1 -- start start 2023-08-25T02:18:49.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a59b2a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:49.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a59b2a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5419a530 con 0x7f1a541067e0 2023-08-25T02:18:49.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:49.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45512/0 (socket says 172.21.15.89:45512) 2023-08-25T02:18:49.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a537fe700 1 -- 172.21.15.89:0/1808909705 learned_addr learned my addr 172.21.15.89:0/1808909705 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:49.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a537fe700 1 -- 172.21.15.89:0/1808909705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48009550 con 0x7f1a541067e0 2023-08-25T02:18:49.111 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.307+0000 7f1a537fe700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fb0 tx=0x7f1a48011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:49.111 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a48003a40 con 0x7f1a541067e0 2023-08-25T02:18:49.111 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a48003ba0 con 0x7f1a541067e0 2023-08-25T02:18:49.111 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5419a730 con 0x7f1a541067e0 2023-08-25T02:18:49.111 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a4801a750 con 0x7f1a541067e0 2023-08-25T02:18:49.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5419ab50 con 0x7f1a541067e0 2023-08-25T02:18:49.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.308+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f1a4801a980 con 0x7f1a541067e0 2023-08-25T02:18:49.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.309+0000 7f1a50ff9700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 0x7f1a3c03a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:49.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.309+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a54061f50 con 0x7f1a541067e0 2023-08-25T02:18:49.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.309+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1a4804c0b0 con 0x7f1a541067e0 2023-08-25T02:18:49.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.309+0000 7f1a52ffd700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 0x7f1a3c03a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:49.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.309+0000 7f1a52ffd700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 0x7f1a3c03a5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1a44006fd0 tx=0x7f1a44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.313+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1a4807ee00 con 0x7f1a541067e0 2023-08-25T02:18:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:46.504+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a54102140 con 0x7f1a3c038110 2023-08-25T02:18:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.769+0000 7f1a50ff9700 1 -- 172.21.15.89:0/1808909705 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1a54102140 con 0x7f1a3c038110 2023-08-25T02:18:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.771+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 msgr2=0x7f1a3c03a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.771+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 0x7f1a3c03a5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1a44006fd0 tx=0x7f1a44006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 msgr2=0x7f1a54199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fb0 tx=0x7f1a48011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 shutdown_connections 2023-08-25T02:18:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f1a3c038110 0x7f1a3c03a5d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 --2- 172.21.15.89:0/1808909705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a541067e0 0x7f1a54199ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:49.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 >> 172.21.15.89:0/1808909705 conn(0x7f1a540fff00 msgr2=0x7f1a54104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:49.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 shutdown_connections 2023-08-25T02:18:49.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:48.772+0000 7f1a59b2a700 1 -- 172.21.15.89:0/1808909705 wait complete. 2023-08-25T02:18:49.116 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mon service... 2023-08-25T02:18:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:49 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:49 smithi089 ceph-mon[98732]: Added host smithi089 2023-08-25T02:18:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:49 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:18:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:49 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:51.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:51 smithi089 ceph-mon[98732]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:51.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:51 smithi089 ceph-mon[98732]: Saving service mon spec with placement count:5 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f005143f700 1 Processor -- start 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f005143f700 1 -- start start 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f005143f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f005143f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c1060f0 con 0x7f004c105700 2023-08-25T02:18:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f004affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:51.346 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f004affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45546/0 (socket says 172.21.15.89:45546) 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.520+0000 7f004affd700 1 -- 172.21.15.89:0/1303349882 learned_addr learned my addr 172.21.15.89:0/1303349882 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f004affd700 1 -- 172.21.15.89:0/1303349882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c106420 con 0x7f004c105700 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f004affd700 1 --2- 172.21.15.89:0/1303349882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f00400098a0 tx=0x7f0040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc89821f5d61424 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f0049ffb700 1 -- 172.21.15.89:0/1303349882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0040004160 con 0x7f004c105700 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f0049ffb700 1 -- 172.21.15.89:0/1303349882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00400042c0 con 0x7f004c105700 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f0049ffb700 1 -- 172.21.15.89:0/1303349882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0040011a00 con 0x7f004c105700 2023-08-25T02:18:51.347 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f005143f700 1 -- 172.21.15.89:0/1303349882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 msgr2=0x7f004c105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f005143f700 1 --2- 172.21.15.89:0/1303349882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f00400098a0 tx=0x7f0040009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f005143f700 1 -- 172.21.15.89:0/1303349882 shutdown_connections 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.521+0000 7f005143f700 1 --2- 172.21.15.89:0/1303349882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c105b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 -- 172.21.15.89:0/1303349882 >> 172.21.15.89:0/1303349882 conn(0x7f004c0fff00 msgr2=0x7f004c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 -- 172.21.15.89:0/1303349882 shutdown_connections 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 -- 172.21.15.89:0/1303349882 wait complete. 2023-08-25T02:18:51.348 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 Processor -- start 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 -- start start 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.522+0000 7f005143f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f005143f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c19cd10 con 0x7f004c105700 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f004affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f004affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45548/0 (socket says 172.21.15.89:45548) 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f004affd700 1 -- 172.21.15.89:0/3904153867 learned_addr learned my addr 172.21.15.89:0/3904153867 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:51.349 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f004affd700 1 -- 172.21.15.89:0/3904153867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040009550 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f004affd700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0040009fb0 tx=0x7f0040011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0040003a40 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0040003ba0 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004c19ce50 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004001a750 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.523+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004c19d210 con 0x7f004c105700 2023-08-25T02:18:51.350 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.524+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f004001a980 con 0x7f004c105700 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.524+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004c061f50 con 0x7f004c105700 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.524+0000 7f0033fff700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 0x7f003403a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.524+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f004004bf20 con 0x7f004c105700 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.524+0000 7f004a7fc700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 0x7f003403a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.525+0000 7f004a7fc700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 0x7f003403a5d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.529+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f004001ac30 con 0x7f004c105700 2023-08-25T02:18:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.746+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f004c109620 con 0x7f0034038110 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.751+0000 7f0033fff700 1 -- 172.21.15.89:0/3904153867 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f004c109620 con 0x7f0034038110 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 msgr2=0x7f003403a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 0x7f003403a5d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 msgr2=0x7f004c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0040009fb0 tx=0x7f0040011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 shutdown_connections 2023-08-25T02:18:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f0034038110 0x7f003403a5d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:51.353 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 --2- 172.21.15.89:0/3904153867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f004c105700 0x7f004c104dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:51.353 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 >> 172.21.15.89:0/3904153867 conn(0x7f004c0fff00 msgr2=0x7f004c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:51.353 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 shutdown_connections 2023-08-25T02:18:51.353 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:49.753+0000 7f005143f700 1 -- 172.21.15.89:0/3904153867 wait complete. 2023-08-25T02:18:51.354 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mgr service... 2023-08-25T02:18:52.591 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-08-25T02:18:52.600 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf92d91700 1 Processor -- start 2023-08-25T02:18:52.600 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf92d91700 1 -- start start 2023-08-25T02:18:52.600 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf92d91700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf92d91700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c1060f0 con 0x7faf8c105700 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf90b2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf90b2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45552/0 (socket says 172.21.15.89:45552) 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.963+0000 7faf90b2d700 1 -- 172.21.15.89:0/823481377 learned_addr learned my addr 172.21.15.89:0/823481377 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.964+0000 7faf90b2d700 1 -- 172.21.15.89:0/823481377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c106420 con 0x7faf8c105700 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.964+0000 7faf90b2d700 1 --2- 172.21.15.89:0/823481377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7faf800098a0 tx=0x7faf80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17759023f0f79fc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:52.601 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.964+0000 7faf8b7fe700 1 -- 172.21.15.89:0/823481377 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf80004160 con 0x7faf8c105700 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.964+0000 7faf8b7fe700 1 -- 172.21.15.89:0/823481377 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf800042c0 con 0x7faf8c105700 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.964+0000 7faf8b7fe700 1 -- 172.21.15.89:0/823481377 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf80011a00 con 0x7faf8c105700 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- 172.21.15.89:0/823481377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 msgr2=0x7faf8c105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 --2- 172.21.15.89:0/823481377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7faf800098a0 tx=0x7faf80009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- 172.21.15.89:0/823481377 shutdown_connections 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 --2- 172.21.15.89:0/823481377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c105b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- 172.21.15.89:0/823481377 >> 172.21.15.89:0/823481377 conn(0x7faf8c0fff00 msgr2=0x7faf8c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:52.602 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- 172.21.15.89:0/823481377 shutdown_connections 2023-08-25T02:18:52.604 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- 172.21.15.89:0/823481377 wait complete. 2023-08-25T02:18:52.604 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 Processor -- start 2023-08-25T02:18:52.604 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- start start 2023-08-25T02:18:52.604 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.965+0000 7faf92d91700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c19c710 con 0x7faf8c105700 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf90b2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf90b2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45554/0 (socket says 172.21.15.89:45554) 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf90b2d700 1 -- 172.21.15.89:0/4202596849 learned_addr learned my addr 172.21.15.89:0/4202596849 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf90b2d700 1 -- 172.21.15.89:0/4202596849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80009550 con 0x7faf8c105700 2023-08-25T02:18:52.605 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf90b2d700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf80009fb0 tx=0x7faf80011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:52.606 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf80003a40 con 0x7faf8c105700 2023-08-25T02:18:52.606 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf80003ba0 con 0x7faf8c105700 2023-08-25T02:18:52.606 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c19c910 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf8001a750 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.966+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c19cd30 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.967+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7faf8001a9d0 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.967+0000 7faf89ffb700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 0x7faf7c03a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.967+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faf8004bfa0 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.967+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c061f50 con 0x7faf8c105700 2023-08-25T02:18:52.607 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.967+0000 7faf8bfff700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 0x7faf7c03a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.968+0000 7faf8bfff700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 0x7faf7c03a580 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf78006fd0 tx=0x7faf78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:51.972+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7faf80023390 con 0x7faf8c105700 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.040+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mon.0 v2:172.21.15.89:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf8002de90 con 0x7faf8c105700 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.187+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7faf8c102110 con 0x7faf7c0380c0 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.192+0000 7faf89ffb700 1 -- 172.21.15.89:0/4202596849 <== mgr.14120 v2:172.21.15.89:6800/3825317465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faf8c102110 con 0x7faf7c0380c0 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.194+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 msgr2=0x7faf7c03a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:52.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.194+0000 7faf92d91700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 0x7faf7c03a580 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf78006fd0 tx=0x7faf78006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.194+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 msgr2=0x7faf8c19c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.194+0000 7faf92d91700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf80009fb0 tx=0x7faf80011ad0 comp rx=0 tx=0).stop 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 shutdown_connections 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7faf7c0380c0 0x7faf7c03a580 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 --2- 172.21.15.89:0/4202596849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf8c105700 0x7faf8c19c1d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 >> 172.21.15.89:0/4202596849 conn(0x7faf8c0fff00 msgr2=0x7faf8c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 shutdown_connections 2023-08-25T02:18:52.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:52.195+0000 7faf92d91700 1 -- 172.21.15.89:0/4202596849 wait complete. 2023-08-25T02:18:53.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:53 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:53 smithi089 ceph-mon[98732]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:18:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:53 smithi089 ceph-mon[98732]: Saving service mgr spec with placement count:2 2023-08-25T02:18:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:53 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:53 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:53.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.032+0000 7fac89be5700 1 Processor -- start 2023-08-25T02:18:53.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac89be5700 1 -- start start 2023-08-25T02:18:53.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac89be5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:53.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac89be5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84104fe0 con 0x7fac841045f0 2023-08-25T02:18:53.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:53.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45558/0 (socket says 172.21.15.89:45558) 2023-08-25T02:18:53.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac837fe700 1 -- 172.21.15.89:0/287319908 learned_addr learned my addr 172.21.15.89:0/287319908 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:53.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac837fe700 1 -- 172.21.15.89:0/287319908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84107580 con 0x7fac841045f0 2023-08-25T02:18:53.868 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac837fe700 1 --2- 172.21.15.89:0/287319908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac780098a0 tx=0x7fac78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82bfb1f1e3b343f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac827fc700 1 -- 172.21.15.89:0/287319908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac78004160 con 0x7fac841045f0 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.033+0000 7fac827fc700 1 -- 172.21.15.89:0/287319908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac780042c0 con 0x7fac841045f0 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac827fc700 1 -- 172.21.15.89:0/287319908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac780119b0 con 0x7fac841045f0 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 -- 172.21.15.89:0/287319908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 msgr2=0x7fac84104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 --2- 172.21.15.89:0/287319908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac780098a0 tx=0x7fac78009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 -- 172.21.15.89:0/287319908 shutdown_connections 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 --2- 172.21.15.89:0/287319908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac84104a10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 -- 172.21.15.89:0/287319908 >> 172.21.15.89:0/287319908 conn(0x7fac840fff00 msgr2=0x7fac84102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 -- 172.21.15.89:0/287319908 shutdown_connections 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.034+0000 7fac89be5700 1 -- 172.21.15.89:0/287319908 wait complete. 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac89be5700 1 Processor -- start 2023-08-25T02:18:53.870 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac89be5700 1 -- start start 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac89be5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac89be5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8419a540 con 0x7fac841045f0 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45560/0 (socket says 172.21.15.89:45560) 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac837fe700 1 -- 172.21.15.89:0/169463949 learned_addr learned my addr 172.21.15.89:0/169463949 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:53.871 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.035+0000 7fac837fe700 1 -- 172.21.15.89:0/169463949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78009550 con 0x7fac841045f0 2023-08-25T02:18:53.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac837fe700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac78003e60 tx=0x7fac78011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:53.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac78003a60 con 0x7fac841045f0 2023-08-25T02:18:53.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac78003bc0 con 0x7fac841045f0 2023-08-25T02:18:53.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8419a740 con 0x7fac841045f0 2023-08-25T02:18:53.872 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac780196c0 con 0x7fac841045f0 2023-08-25T02:18:53.880 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.036+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8419ab60 con 0x7fac841045f0 2023-08-25T02:18:53.880 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7fac78019930 con 0x7fac841045f0 2023-08-25T02:18:53.880 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac84061f50 con 0x7fac841045f0 2023-08-25T02:18:53.880 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac80ff9700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 0x7fac6c03a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:53.880 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac7804c2f0 con 0x7fac841045f0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac82ffd700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 0x7fac6c03a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.037+0000 7fac82ffd700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 0x7fac6c03a600 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac74006fd0 tx=0x7fac74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.042+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fac78010de0 con 0x7fac841045f0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.247+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fac8410a780 con 0x7fac841045f0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.250+0000 7fac80ff9700 1 -- 172.21.15.89:0/169463949 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fac78010380 con 0x7fac841045f0 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.252+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 msgr2=0x7fac6c03a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.252+0000 7fac89be5700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 0x7fac6c03a600 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac74006fd0 tx=0x7fac74006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.252+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 msgr2=0x7fac8419a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.252+0000 7fac89be5700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac78003e60 tx=0x7fac78011a30 comp rx=0 tx=0).stop 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.253+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 shutdown_connections 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.253+0000 7fac89be5700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7fac6c038140 0x7fac6c03a600 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.254+0000 7fac89be5700 1 --2- 172.21.15.89:0/169463949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac841045f0 0x7fac8419a000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.254+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 >> 172.21.15.89:0/169463949 conn(0x7fac840fff00 msgr2=0x7fac84100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:53.882 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.254+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 shutdown_connections 2023-08-25T02:18:53.883 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:53.254+0000 7fac89be5700 1 -- 172.21.15.89:0/169463949 wait complete. 2023-08-25T02:18:54.265 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:54 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/169463949' entity='client.admin' 2023-08-25T02:18:54.265 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:54 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:54.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.318+0000 7f7ceb48e700 1 Processor -- start 2023-08-25T02:18:54.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.318+0000 7f7ceb48e700 1 -- start start 2023-08-25T02:18:54.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.318+0000 7f7ceb48e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:54.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.318+0000 7f7ceb48e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4105be0 con 0x7f7ce41051f0 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7ce922a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7ce922a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45564/0 (socket says 172.21.15.89:45564) 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7ce922a700 1 -- 172.21.15.89:0/3614559769 learned_addr learned my addr 172.21.15.89:0/3614559769 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7ce922a700 1 -- 172.21.15.89:0/3614559769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce41063f0 con 0x7f7ce41051f0 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7ce922a700 1 --2- 172.21.15.89:0/3614559769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00098a0 tx=0x7f7ce0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f1aefa0c68db4e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7cdbfff700 1 -- 172.21.15.89:0/3614559769 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0004160 con 0x7f7ce41051f0 2023-08-25T02:18:54.888 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7cdbfff700 1 -- 172.21.15.89:0/3614559769 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce00042c0 con 0x7f7ce41051f0 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.319+0000 7f7cdbfff700 1 -- 172.21.15.89:0/3614559769 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce00119b0 con 0x7f7ce41051f0 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 -- 172.21.15.89:0/3614559769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 msgr2=0x7f7ce4105610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/3614559769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00098a0 tx=0x7f7ce0009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 -- 172.21.15.89:0/3614559769 shutdown_connections 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/3614559769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4105610 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 -- 172.21.15.89:0/3614559769 >> 172.21.15.89:0/3614559769 conn(0x7f7ce4100770 msgr2=0x7f7ce4102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 -- 172.21.15.89:0/3614559769 shutdown_connections 2023-08-25T02:18:54.889 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 -- 172.21.15.89:0/3614559769 wait complete. 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.320+0000 7f7ceb48e700 1 Processor -- start 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ceb48e700 1 -- start start 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ceb48e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ceb48e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4199420 con 0x7f7ce41051f0 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ce922a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ce922a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45566/0 (socket says 172.21.15.89:45566) 2023-08-25T02:18:54.890 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ce922a700 1 -- 172.21.15.89:0/1104601264 learned_addr learned my addr 172.21.15.89:0/1104601264 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:54.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ce922a700 1 -- 172.21.15.89:0/1104601264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0009550 con 0x7f7ce41051f0 2023-08-25T02:18:54.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ce922a700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006af0 tx=0x7f7ce0011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:54.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0003a40 con 0x7f7ce41051f0 2023-08-25T02:18:54.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce4199620 con 0x7f7ce41051f0 2023-08-25T02:18:54.891 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce0003ba0 con 0x7f7ce41051f0 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.321+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0019690 con 0x7f7ce41051f0 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.322+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce4199980 con 0x7f7ce41051f0 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.322+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f7ce0019950 con 0x7f7ce41051f0 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.322+0000 7f7cda7fc700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 0x7f7cd003a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.322+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7ce004bed0 con 0x7f7ce41051f0 2023-08-25T02:18:54.892 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.323+0000 7f7ce8a29700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 0x7f7cd003a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.323+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4061f50 con 0x7f7ce41051f0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.323+0000 7f7ce8a29700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 0x7f7cd003a580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006fd0 tx=0x7f7cd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.328+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7ce0018350 con 0x7f7ce41051f0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.544+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7ce41095f0 con 0x7f7ce41051f0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.550+0000 7f7cda7fc700 1 -- 172.21.15.89:0/1104601264 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7ce0049020 con 0x7f7ce41051f0 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 msgr2=0x7f7cd003a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:54.893 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 0x7f7cd003a580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006fd0 tx=0x7f7cd4006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:54.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 msgr2=0x7f7ce4198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:54.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006af0 tx=0x7f7ce0011a10 comp rx=0 tx=0).stop 2023-08-25T02:18:54.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 shutdown_connections 2023-08-25T02:18:54.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f7cd00380c0 0x7f7cd003a580 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:54.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 --2- 172.21.15.89:0/1104601264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ce41051f0 0x7f7ce4198ee0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:54.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.552+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 >> 172.21.15.89:0/1104601264 conn(0x7f7ce4100770 msgr2=0x7f7ce4101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:54.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.553+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 shutdown_connections 2023-08-25T02:18:54.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:54.553+0000 7f7ceb48e700 1 -- 172.21.15.89:0/1104601264 wait complete. 2023-08-25T02:18:54.896 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling the dashboard module... 2023-08-25T02:18:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:55 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1104601264' entity='client.admin' 2023-08-25T02:18:55.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:55 smithi089 ceph-mon[98732]: from='mgr.14120 172.21.15.89:0/3483124656' entity='mgr.x' 2023-08-25T02:18:56.819 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3061127083' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-08-25T02:18:56.819 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setuser ceph since I am not root 2023-08-25T02:18:56.820 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setgroup ceph since I am not root 2023-08-25T02:18:56.820 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:56.738+0000 7fa1e8422200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:18:57.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.305+0000 7f358260b700 1 Processor -- start 2023-08-25T02:18:57.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.305+0000 7f358260b700 1 -- start start 2023-08-25T02:18:57.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.305+0000 7f358260b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:57.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.305+0000 7f358260b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c079e70 con 0x7f357c106d90 2023-08-25T02:18:57.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.305+0000 7f357bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:57.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45570/0 (socket says 172.21.15.89:45570) 2023-08-25T02:18:57.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357bfff700 1 -- 172.21.15.89:0/3404860036 learned_addr learned my addr 172.21.15.89:0/3404860036 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:57.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357bfff700 1 -- 172.21.15.89:0/3404860036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c1096c0 con 0x7f357c106d90 2023-08-25T02:18:57.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357bfff700 1 --2- 172.21.15.89:0/3404860036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35640098a0 tx=0x7f3564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a459bf336ecb5529 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:57.048 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357affd700 1 -- 172.21.15.89:0/3404860036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3564004160 con 0x7f357c106d90 2023-08-25T02:18:57.048 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357affd700 1 -- 172.21.15.89:0/3404860036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35640042c0 con 0x7f357c106d90 2023-08-25T02:18:57.049 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.306+0000 7f357affd700 1 -- 172.21.15.89:0/3404860036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35640119b0 con 0x7f357c106d90 2023-08-25T02:18:57.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- 172.21.15.89:0/3404860036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 msgr2=0x7f357c109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:57.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 --2- 172.21.15.89:0/3404860036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35640098a0 tx=0x7f3564009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:57.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- 172.21.15.89:0/3404860036 shutdown_connections 2023-08-25T02:18:57.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 --2- 172.21.15.89:0/3404860036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c109180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:57.053 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- 172.21.15.89:0/3404860036 >> 172.21.15.89:0/3404860036 conn(0x7f357c1007c0 msgr2=0x7f357c102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- 172.21.15.89:0/3404860036 shutdown_connections 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- 172.21.15.89:0/3404860036 wait complete. 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 Processor -- start 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.307+0000 7f358260b700 1 -- start start 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f358260b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f358260b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c19a500 con 0x7f357c106d90 2023-08-25T02:18:57.054 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f357bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:57.055 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f357bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45572/0 (socket says 172.21.15.89:45572) 2023-08-25T02:18:57.057 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f357bfff700 1 -- 172.21.15.89:0/3061127083 learned_addr learned my addr 172.21.15.89:0/3061127083 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:57.058 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f357bfff700 1 -- 172.21.15.89:0/3061127083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3564009550 con 0x7f357c106d90 2023-08-25T02:18:57.058 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.308+0000 7f357bfff700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3564006af0 tx=0x7f3564011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:57.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3564003a40 con 0x7f357c106d90 2023-08-25T02:18:57.059 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3564003ba0 con 0x7f357c106d90 2023-08-25T02:18:57.064 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357c19a700 con 0x7f357c106d90 2023-08-25T02:18:57.064 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35640196c0 con 0x7f357c106d90 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357c19ab20 con 0x7f357c106d90 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.309+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 45135+0+0 (secure 0 0 0) 0x7f356401a430 con 0x7f357c106d90 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.310+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c061f50 con 0x7f357c106d90 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.310+0000 7f35797fa700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 0x7f356803a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.310+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f356404c080 con 0x7f357c106d90 2023-08-25T02:18:57.065 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.310+0000 7f357b7fe700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 0x7f356803a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:57.066 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.310+0000 7f357b7fe700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 0x7f356803a580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f356c006fd0 tx=0x7f356c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:57.068 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.314+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f356407ee00 con 0x7f357c106d90 2023-08-25T02:18:57.068 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:55.575+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f357c10a4a0 con 0x7f357c106d90 2023-08-25T02:18:57.068 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.556+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 9) v1 ==== 45148+0+0 (secure 0 0 0) 0x7f356400bce0 con 0x7f357c106d90 2023-08-25T02:18:57.069 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.557+0000 7f35797fa700 1 -- 172.21.15.89:0/3061127083 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f356401a6e0 con 0x7f357c106d90 2023-08-25T02:18:57.069 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 msgr2=0x7f356803a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:57.069 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 0x7f356803a580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f356c006fd0 tx=0x7f356c006e40 comp rx=0 tx=0).stop 2023-08-25T02:18:57.070 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 msgr2=0x7f357c199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:57.071 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3564006af0 tx=0x7f3564011a10 comp rx=0 tx=0).stop 2023-08-25T02:18:57.071 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 shutdown_connections 2023-08-25T02:18:57.071 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f35680380c0 0x7f356803a580 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:57.071 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 --2- 172.21.15.89:0/3061127083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f357c106d90 0x7f357c199fc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:57.071 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 >> 172.21.15.89:0/3061127083 conn(0x7f357c1007c0 msgr2=0x7f357c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:57.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 shutdown_connections 2023-08-25T02:18:57.072 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:56.560+0000 7f358260b700 1 -- 172.21.15.89:0/3061127083 wait complete. 2023-08-25T02:18:57.085 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:56.818+0000 7fa1e8422200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:18:57.085 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:56.908+0000 7fa1e8422200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:18:57.086 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.004+0000 7fa1e8422200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:18:57.558 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.189+0000 7fa1e8422200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:18:57.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3061127083' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-08-25T02:18:57.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-mon[98732]: mgrmap e9: x(active, since 16s) 2023-08-25T02:18:57.815 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.579+0000 7fa1e8422200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:18:57.816 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.712+0000 7fa1e8422200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:18:58.082 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.808+0000 7fa1e8422200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:18:58.082 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:57.908+0000 7fa1e8422200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:18:58.098 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:18:58.098 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-08-25T02:18:58.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2023-08-25T02:18:58.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-08-25T02:18:58.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-08-25T02:18:58.112 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:18:58.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f9392700 1 Processor -- start 2023-08-25T02:18:58.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f9392700 1 -- start start 2023-08-25T02:18:58.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f9392700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:58.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f9392700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4074120 con 0x7f18f4106da0 2023-08-25T02:18:58.113 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45586/0 (socket says 172.21.15.89:45586) 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f2ffd700 1 -- 172.21.15.89:0/459515602 learned_addr learned my addr 172.21.15.89:0/459515602 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f2ffd700 1 -- 172.21.15.89:0/459515602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f41096d0 con 0x7f18f4106da0 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.453+0000 7f18f2ffd700 1 --2- 172.21.15.89:0/459515602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18e40098a0 tx=0x7f18e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d346647217e41e8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f1ffb700 1 -- 172.21.15.89:0/459515602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4004160 con 0x7f18f4106da0 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f1ffb700 1 -- 172.21.15.89:0/459515602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e40042c0 con 0x7f18f4106da0 2023-08-25T02:18:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f1ffb700 1 -- 172.21.15.89:0/459515602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e40119b0 con 0x7f18f4106da0 2023-08-25T02:18:58.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 -- 172.21.15.89:0/459515602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 msgr2=0x7f18f4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:58.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 --2- 172.21.15.89:0/459515602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18e40098a0 tx=0x7f18e4009bb0 comp rx=0 tx=0).stop 2023-08-25T02:18:58.115 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 -- 172.21.15.89:0/459515602 shutdown_connections 2023-08-25T02:18:58.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 --2- 172.21.15.89:0/459515602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4109190 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:58.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 -- 172.21.15.89:0/459515602 >> 172.21.15.89:0/459515602 conn(0x7f18f4100770 msgr2=0x7f18f4102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:58.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 -- 172.21.15.89:0/459515602 shutdown_connections 2023-08-25T02:18:58.116 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.454+0000 7f18f9392700 1 -- 172.21.15.89:0/459515602 wait complete. 2023-08-25T02:18:58.117 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f9392700 1 Processor -- start 2023-08-25T02:18:58.117 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f9392700 1 -- start start 2023-08-25T02:18:58.118 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f9392700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:58.118 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f9392700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f419a500 con 0x7f18f4106da0 2023-08-25T02:18:58.118 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:18:58.119 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45588/0 (socket says 172.21.15.89:45588) 2023-08-25T02:18:58.119 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.455+0000 7f18f2ffd700 1 -- 172.21.15.89:0/2654372881 learned_addr learned my addr 172.21.15.89:0/2654372881 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:18:58.119 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18f2ffd700 1 -- 172.21.15.89:0/2654372881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e4009550 con 0x7f18f4106da0 2023-08-25T02:18:58.119 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18f2ffd700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18e4003f00 tx=0x7f18e4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:18:58.119 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4003b20 con 0x7f18f4106da0 2023-08-25T02:18:58.131 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e4003c80 con 0x7f18f4106da0 2023-08-25T02:18:58.131 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f419a700 con 0x7f18f4106da0 2023-08-25T02:18:58.131 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4019770 con 0x7f18f4106da0 2023-08-25T02:18:58.131 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.456+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f419ab20 con 0x7f18f4106da0 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.457+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f4061f50 con 0x7f18f4106da0 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.457+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 45148+0+0 (secure 0 0 0) 0x7f18e401ab60 con 0x7f18f4106da0 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.458+0000 7f18dbfff700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 0x7f18dc03a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.458+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f18e404c430 con 0x7f18f4106da0 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.458+0000 7f18f27fc700 1 -- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 msgr2=0x7f18dc03a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.458+0000 7f18f27fc700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 0x7f18dc03a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:18:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.466+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f18e401a430 con 0x7f18f4106da0 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.659+0000 7f18f27fc700 1 -- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 msgr2=0x7f18dc03a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.659+0000 7f18f27fc700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 0x7f18dc03a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.723+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f18f410a6e0 con 0x7f18f4106da0 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.724+0000 7f18dbfff700 1 -- 172.21.15.89:0/2654372881 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f18e402a070 con 0x7f18f4106da0 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 msgr2=0x7f18dc03a620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 0x7f18dc03a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:58.133 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 msgr2=0x7f18f4199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:18:58.134 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18e4003f00 tx=0x7f18e4011a30 comp rx=0 tx=0).stop 2023-08-25T02:18:58.134 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 shutdown_connections 2023-08-25T02:18:58.135 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f18dc038160 0x7f18dc03a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:58.135 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 --2- 172.21.15.89:0/2654372881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18f4106da0 0x7f18f4199fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:18:58.135 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 >> 172.21.15.89:0/2654372881 conn(0x7f18f4100770 msgr2=0x7f18f4101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:18:58.135 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 shutdown_connections 2023-08-25T02:18:58.136 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:57.726+0000 7f18f9392700 1 -- 172.21.15.89:0/2654372881 wait complete. 2023-08-25T02:18:58.136 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2023-08-25T02:18:58.136 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 9... 2023-08-25T02:18:58.556 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:58.347+0000 7fa1e8422200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:18:58.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:18:58 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2654372881' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-08-25T02:18:59.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:59.007+0000 7fa1e8422200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:18:59.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:59.089+0000 7fa1e8422200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:19:00.116 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:59.853+0000 7fa1e8422200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:19:00.116 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:18:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:18:59.943+0000 7fa1e8422200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:19:00.394 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:00.115+0000 7fa1e8422200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:19:00.394 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:00.393+0000 7fa1e8422200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:19:00.762 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:00.614+0000 7fa1e8422200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:19:01.334 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:00.761+0000 7fa1e8422200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:19:01.334 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:01.021+0000 7fa1e8422200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:19:02.038 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:01.711+0000 7fa1e8422200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:19:02.039 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:01.841+0000 7fa1e8422200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:19:02.039 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:02.033+0000 7fa1e8422200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:19:02.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:02.145+0000 7fa1e8422200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:19:02.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:02.361+0000 7fa1e8422200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:19:03.580 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:03.207+0000 7fa1e8422200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:19:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: Active manager daemon x restarted 2023-08-25T02:19:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: Activating manager daemon x 2023-08-25T02:19:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: osdmap e3: 0 total, 0 up, 0 in 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: mgrmap e10: x(active, starting, since 0.0036518s) 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: Manager daemon x is now available 2023-08-25T02:19:03.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:03.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:04.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:19:04.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:19:04.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:04 smithi089 ceph-mon[98732]: mgrmap e11: x(active, since 1.00553s) 2023-08-25T02:19:04.846 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2023-08-25T02:19:04.846 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-08-25T02:19:04.846 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2023-08-25T02:19:04.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2023-08-25T02:19:04.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.587+0000 7f2d8b0b1700 1 Processor -- start 2023-08-25T02:19:04.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.587+0000 7f2d8b0b1700 1 -- start start 2023-08-25T02:19:04.847 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.587+0000 7f2d8b0b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:04.848 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.587+0000 7f2d8b0b1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840740e0 con 0x7f2d84104bd0 2023-08-25T02:19:04.849 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d88e4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:04.849 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d88e4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45596/0 (socket says 172.21.15.89:45596) 2023-08-25T02:19:04.849 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d88e4d700 1 -- 172.21.15.89:0/1593067847 learned_addr learned my addr 172.21.15.89:0/1593067847 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:04.850 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d88e4d700 1 -- 172.21.15.89:0/1593067847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84105530 con 0x7f2d84104bd0 2023-08-25T02:19:04.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d88e4d700 1 --2- 172.21.15.89:0/1593067847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d780098a0 tx=0x7f2d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3285a9ab03c277 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:04.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d837fe700 1 -- 172.21.15.89:0/1593067847 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d78004160 con 0x7f2d84104bd0 2023-08-25T02:19:04.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d837fe700 1 -- 172.21.15.89:0/1593067847 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d780042c0 con 0x7f2d84104bd0 2023-08-25T02:19:04.851 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.588+0000 7f2d837fe700 1 -- 172.21.15.89:0/1593067847 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d780119b0 con 0x7f2d84104bd0 2023-08-25T02:19:04.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/1593067847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 msgr2=0x7f2d84104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:04.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/1593067847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d780098a0 tx=0x7f2d78009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.852 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/1593067847 shutdown_connections 2023-08-25T02:19:04.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/1593067847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d84104ff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.853 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/1593067847 >> 172.21.15.89:0/1593067847 conn(0x7f2d84100770 msgr2=0x7f2d84102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/1593067847 shutdown_connections 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/1593067847 wait complete. 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 Processor -- start 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.589+0000 7f2d8b0b1700 1 -- start start 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d8b0b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:04.854 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d8b0b1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8419c710 con 0x7f2d84104bd0 2023-08-25T02:19:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d88e4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d88e4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45598/0 (socket says 172.21.15.89:45598) 2023-08-25T02:19:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d88e4d700 1 -- 172.21.15.89:0/3219774533 learned_addr learned my addr 172.21.15.89:0/3219774533 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:04.855 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d88e4d700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78009550 con 0x7f2d84104bd0 2023-08-25T02:19:04.856 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d88e4d700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2d78003f00 tx=0x7f2d78011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:04.856 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d78003b20 con 0x7f2d84104bd0 2023-08-25T02:19:04.856 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d78003c80 con 0x7f2d84104bd0 2023-08-25T02:19:04.857 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d78019770 con 0x7f2d84104bd0 2023-08-25T02:19:04.857 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.590+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8419c910 con 0x7f2d84104bd0 2023-08-25T02:19:04.857 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.591+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8419cd30 con 0x7f2d84104bd0 2023-08-25T02:19:04.857 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.591+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 45148+0+0 (secure 0 0 0) 0x7f2d7801ab60 con 0x7f2d84104bd0 2023-08-25T02:19:04.858 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.592+0000 7f2d81ffb700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:04.858 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.592+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d7403ad30 con 0x7f2d74038160 2023-08-25T02:19:04.858 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.592+0000 7f2d83fff700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:19:04.859 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.592+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2d7804c240 con 0x7f2d84104bd0 2023-08-25T02:19:04.859 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.592+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:19:04.859 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.793+0000 7f2d83fff700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:19:04.859 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:58.793+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:19:04.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:59.194+0000 7f2d83fff700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:19:04.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:59.194+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T02:19:04.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:59.995+0000 7f2d83fff700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:19:04.860 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:18:59.995+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T02:19:04.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:01.596+0000 7f2d83fff700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3825317465 2023-08-25T02:19:04.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:01.596+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T02:19:04.861 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:03.222+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 10) v1 ==== 44915+0+0 (secure 0 0 0) 0x7f2d78019a30 con 0x7f2d84104bd0 2023-08-25T02:19:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:03.222+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 msgr2=0x7f2d7403a620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:19:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:03.222+0000 7f2d81ffb700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.223+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 11) v1 ==== 45042+0+0 (secure 0 0 0) 0x7f2d7802ad30 con 0x7f2d84104bd0 2023-08-25T02:19:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.224+0000 7f2d81ffb700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 0x7f2d7403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.224+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d7403ad30 con 0x7f2d7403bc10 2023-08-25T02:19:04.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.224+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 0x7f2d7403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:04.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.224+0000 7f2d83fff700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 0x7f2d7403e000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d70003a10 tx=0x7f2d700092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:04.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.225+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f2d7403ad30 con 0x7f2d7403bc10 2023-08-25T02:19:04.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.231+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2d8419d510 con 0x7f2d7403bc10 2023-08-25T02:19:04.863 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.231+0000 7f2d81ffb700 1 -- 172.21.15.89:0/3219774533 <== mgr.14152 v2:172.21.15.89:6800/1949193934 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f2d8419d510 con 0x7f2d7403bc10 2023-08-25T02:19:04.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 msgr2=0x7f2d7403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:04.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 0x7f2d7403e000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d70003a10 tx=0x7f2d700092b0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 msgr2=0x7f2d8419c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:04.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2d78003f00 tx=0x7f2d78011a30 comp rx=0 tx=0).stop 2023-08-25T02:19:04.864 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 shutdown_connections 2023-08-25T02:19:04.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2d7403bc10 0x7f2d7403e000 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84104bd0 0x7f2d8419c1d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 --2- 172.21.15.89:0/3219774533 >> [v2:172.21.15.89:6800/3825317465,v1:172.21.15.89:6801/3825317465] conn(0x7f2d74038160 0x7f2d7403a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:04.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 >> 172.21.15.89:0/3219774533 conn(0x7f2d84100770 msgr2=0x7f2d84190790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:04.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 shutdown_connections 2023-08-25T02:19:04.866 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:04.232+0000 7f2d8b0b1700 1 -- 172.21.15.89:0/3219774533 wait complete. 2023-08-25T02:19:04.866 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 9 is available 2023-08-25T02:19:04.867 INFO:teuthology.orchestra.run.smithi089.stdout:Generating a dashboard self-signed certificate... 2023-08-25T02:19:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: [25/Aug/2023:02:19:03] ENGINE Bus STARTING 2023-08-25T02:19:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: [25/Aug/2023:02:19:03] ENGINE Serving on http://172.21.15.89:8765 2023-08-25T02:19:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: [25/Aug/2023:02:19:03] ENGINE Serving on https://172.21.15.89:7150 2023-08-25T02:19:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: [25/Aug/2023:02:19:03] ENGINE Bus STARTED 2023-08-25T02:19:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-08-25T02:19:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-08-25T02:19:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:05.984 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-08-25T02:19:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf43325700 1 Processor -- start 2023-08-25T02:19:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf43325700 1 -- start start 2023-08-25T02:19:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf43325700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf43325700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c0740e0 con 0x7fcf3c106de0 2023-08-25T02:19:05.985 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf410c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf410c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45652/0 (socket says 172.21.15.89:45652) 2023-08-25T02:19:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.244+0000 7fcf410c1700 1 -- 172.21.15.89:0/1206779732 learned_addr learned my addr 172.21.15.89:0/1206779732 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf410c1700 1 -- 172.21.15.89:0/1206779732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c109710 con 0x7fcf3c106de0 2023-08-25T02:19:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf410c1700 1 --2- 172.21.15.89:0/1206779732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf380098a0 tx=0x7fcf38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bb74c4f4f14a5c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf33fff700 1 -- 172.21.15.89:0/1206779732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf38004160 con 0x7fcf3c106de0 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf33fff700 1 -- 172.21.15.89:0/1206779732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf380042c0 con 0x7fcf3c106de0 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf33fff700 1 -- 172.21.15.89:0/1206779732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf380119b0 con 0x7fcf3c106de0 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf43325700 1 -- 172.21.15.89:0/1206779732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 msgr2=0x7fcf3c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.245+0000 7fcf43325700 1 --2- 172.21.15.89:0/1206779732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf380098a0 tx=0x7fcf38009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- 172.21.15.89:0/1206779732 shutdown_connections 2023-08-25T02:19:05.987 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 --2- 172.21.15.89:0/1206779732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c1091d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:05.988 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- 172.21.15.89:0/1206779732 >> 172.21.15.89:0/1206779732 conn(0x7fcf3c100770 msgr2=0x7fcf3c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:05.988 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- 172.21.15.89:0/1206779732 shutdown_connections 2023-08-25T02:19:05.989 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- 172.21.15.89:0/1206779732 wait complete. 2023-08-25T02:19:05.989 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 Processor -- start 2023-08-25T02:19:05.989 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- start start 2023-08-25T02:19:05.989 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:05.990 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf43325700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c19a510 con 0x7fcf3c106de0 2023-08-25T02:19:05.990 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf410c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:05.990 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf410c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45654/0 (socket says 172.21.15.89:45654) 2023-08-25T02:19:05.990 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.246+0000 7fcf410c1700 1 -- 172.21.15.89:0/1942693835 learned_addr learned my addr 172.21.15.89:0/1942693835 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf410c1700 1 -- 172.21.15.89:0/1942693835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38009550 con 0x7fcf3c106de0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf410c1700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf38006af0 tx=0x7fcf38011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf38003a40 con 0x7fcf3c106de0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c19a710 con 0x7fcf3c106de0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf38003ba0 con 0x7fcf3c106de0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf380196c0 con 0x7fcf3c106de0 2023-08-25T02:19:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.247+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c19ab30 con 0x7fcf3c106de0 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 45042+0+0 (secure 0 0 0) 0x7fcf38019980 con 0x7fcf3c106de0 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3c061f50 con 0x7fcf3c106de0 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf327fc700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 0x7fcf2803a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcf3804c380 con 0x7fcf3c106de0 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf408c0700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 0x7fcf2803a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:05.992 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.248+0000 7fcf408c0700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 0x7fcf2803a500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006fd0 tx=0x7fcf2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.253+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fcf38023d20 con 0x7fcf3c106de0 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.471+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf3c102a80 con 0x7fcf28038040 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.609+0000 7fcf327fc700 1 -- 172.21.15.89:0/1942693835 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcf3c102a80 con 0x7fcf28038040 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 msgr2=0x7fcf2803a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 0x7fcf2803a500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006fd0 tx=0x7fcf2c006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 msgr2=0x7fcf3c199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf38006af0 tx=0x7fcf38011a10 comp rx=0 tx=0).stop 2023-08-25T02:19:05.993 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 shutdown_connections 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcf28038040 0x7fcf2803a500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 --2- 172.21.15.89:0/1942693835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf3c106de0 0x7fcf3c199fd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 >> 172.21.15.89:0/1942693835 conn(0x7fcf3c100770 msgr2=0x7fcf3c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 shutdown_connections 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:05.612+0000 7fcf43325700 1 -- 172.21.15.89:0/1942693835 wait complete. 2023-08-25T02:19:05.994 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial admin user... 2023-08-25T02:19:06.877 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:06 smithi089 ceph-mon[98732]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:06.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:06.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:06.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:06.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:06 smithi089 ceph-mon[98732]: mgrmap e12: x(active, since 2s) 2023-08-25T02:19:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$5OH0Z1LWgYr/kdWjqWnj6efUtVi7U0EFbDrbE7S549ZtKxCxqCPkS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1692929946, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-08-25T02:19:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.429+0000 7fcee0076700 1 Processor -- start 2023-08-25T02:19:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcee0076700 1 -- start start 2023-08-25T02:19:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcee0076700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcee0076700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced8105be0 con 0x7fced81051f0 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcedde12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcedde12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45658/0 (socket says 172.21.15.89:45658) 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcedde12700 1 -- 172.21.15.89:0/4093207314 learned_addr learned my addr 172.21.15.89:0/4093207314 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcedde12700 1 -- 172.21.15.89:0/4093207314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced81063f0 con 0x7fced81051f0 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.430+0000 7fcedde12700 1 --2- 172.21.15.89:0/4093207314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0098a0 tx=0x7fcecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5576eb3c0ec1119b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcedce10700 1 -- 172.21.15.89:0/4093207314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc004160 con 0x7fced81051f0 2023-08-25T02:19:07.539 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcedce10700 1 -- 172.21.15.89:0/4093207314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcecc0042c0 con 0x7fced81051f0 2023-08-25T02:19:07.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcedce10700 1 -- 172.21.15.89:0/4093207314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc0119b0 con 0x7fced81051f0 2023-08-25T02:19:07.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 -- 172.21.15.89:0/4093207314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 msgr2=0x7fced8105610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:07.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 --2- 172.21.15.89:0/4093207314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0098a0 tx=0x7fcecc009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:07.559 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 -- 172.21.15.89:0/4093207314 shutdown_connections 2023-08-25T02:19:07.559 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 --2- 172.21.15.89:0/4093207314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced8105610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:07.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 -- 172.21.15.89:0/4093207314 >> 172.21.15.89:0/4093207314 conn(0x7fced8100770 msgr2=0x7fced8102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:07.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 -- 172.21.15.89:0/4093207314 shutdown_connections 2023-08-25T02:19:07.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.431+0000 7fcee0076700 1 -- 172.21.15.89:0/4093207314 wait complete. 2023-08-25T02:19:07.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcee0076700 1 Processor -- start 2023-08-25T02:19:07.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcee0076700 1 -- start start 2023-08-25T02:19:07.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcee0076700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcee0076700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced819a570 con 0x7fced81051f0 2023-08-25T02:19:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcedde12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcedde12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45660/0 (socket says 172.21.15.89:45660) 2023-08-25T02:19:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcedde12700 1 -- 172.21.15.89:0/2231709194 learned_addr learned my addr 172.21.15.89:0/2231709194 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcedde12700 1 -- 172.21.15.89:0/2231709194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc009550 con 0x7fced81051f0 2023-08-25T02:19:07.563 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.432+0000 7fcedde12700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006af0 tx=0x7fcecc011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:07.563 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc003a40 con 0x7fced81051f0 2023-08-25T02:19:07.563 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcecc003ba0 con 0x7fced81051f0 2023-08-25T02:19:07.564 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced819a770 con 0x7fced81051f0 2023-08-25T02:19:07.565 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc019690 con 0x7fced81051f0 2023-08-25T02:19:07.565 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced819ab90 con 0x7fced81051f0 2023-08-25T02:19:07.565 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.433+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 45148+0+0 (secure 0 0 0) 0x7fcecc019950 con 0x7fced81051f0 2023-08-25T02:19:07.565 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.434+0000 7fcecaffd700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 0x7fcec403a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:07.566 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.434+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcecc04c4f0 con 0x7fced81051f0 2023-08-25T02:19:07.586 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.434+0000 7fcedd611700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 0x7fcec403a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:07.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.434+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced8061f50 con 0x7fced81051f0 2023-08-25T02:19:07.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.434+0000 7fcedd611700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 0x7fcec403a5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fced4006fd0 tx=0x7fced4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:07.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.438+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fcecc0243f0 con 0x7fced81051f0 2023-08-25T02:19:07.587 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.650+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fced8061d70 con 0x7fcec4038110 2023-08-25T02:19:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.877+0000 7fcecaffd700 1 -- 172.21.15.89:0/2231709194 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fced8061d70 con 0x7fcec4038110 2023-08-25T02:19:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 msgr2=0x7fcec403a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 0x7fcec403a5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fced4006fd0 tx=0x7fced4006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 msgr2=0x7fced819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006af0 tx=0x7fcecc011a10 comp rx=0 tx=0).stop 2023-08-25T02:19:07.596 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 shutdown_connections 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcec4038110 0x7fcec403a5d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 --2- 172.21.15.89:0/2231709194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fced81051f0 0x7fced819a030 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 >> 172.21.15.89:0/2231709194 conn(0x7fced8100770 msgr2=0x7fced8101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 shutdown_connections 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:06.880+0000 7fcee0076700 1 -- 172.21.15.89:0/2231709194 wait complete. 2023-08-25T02:19:07.597 INFO:teuthology.orchestra.run.smithi089.stdout:Fetching dashboard port number... 2023-08-25T02:19:08.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:07 smithi089 ceph-mon[98732]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:08.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:08.494 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 8443 2023-08-25T02:19:08.494 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.964+0000 7fa6b9cbd700 1 Processor -- start 2023-08-25T02:19:08.494 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.964+0000 7fa6b9cbd700 1 -- start start 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.964+0000 7fa6b9cbd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.964+0000 7fa6b9cbd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4074120 con 0x7fa6b4106da0 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45664/0 (socket says 172.21.15.89:45664) 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b37fe700 1 -- 172.21.15.89:0/1180951621 learned_addr learned my addr 172.21.15.89:0/1180951621 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b37fe700 1 -- 172.21.15.89:0/1180951621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b41096d0 con 0x7fa6b4106da0 2023-08-25T02:19:08.495 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b37fe700 1 --2- 172.21.15.89:0/1180951621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80098a0 tx=0x7fa6a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b6f97a1290e796 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:08.496 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b27fc700 1 -- 172.21.15.89:0/1180951621 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6a8004160 con 0x7fa6b4106da0 2023-08-25T02:19:08.496 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b27fc700 1 -- 172.21.15.89:0/1180951621 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6a80042c0 con 0x7fa6b4106da0 2023-08-25T02:19:08.496 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.965+0000 7fa6b27fc700 1 -- 172.21.15.89:0/1180951621 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6a80119b0 con 0x7fa6b4106da0 2023-08-25T02:19:08.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/1180951621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 msgr2=0x7fa6b4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:08.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/1180951621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80098a0 tx=0x7fa6a8009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:08.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/1180951621 shutdown_connections 2023-08-25T02:19:08.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/1180951621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4109190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:08.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/1180951621 >> 172.21.15.89:0/1180951621 conn(0x7fa6b4100770 msgr2=0x7fa6b4102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:08.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/1180951621 shutdown_connections 2023-08-25T02:19:08.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/1180951621 wait complete. 2023-08-25T02:19:08.498 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.966+0000 7fa6b9cbd700 1 Processor -- start 2023-08-25T02:19:08.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b9cbd700 1 -- start start 2023-08-25T02:19:08.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b9cbd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:08.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b9cbd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b419a500 con 0x7fa6b4106da0 2023-08-25T02:19:08.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:08.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45666/0 (socket says 172.21.15.89:45666) 2023-08-25T02:19:08.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b37fe700 1 -- 172.21.15.89:0/701897403 learned_addr learned my addr 172.21.15.89:0/701897403 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:08.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b37fe700 1 -- 172.21.15.89:0/701897403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a8009550 con 0x7fa6b4106da0 2023-08-25T02:19:08.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b37fe700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8006af0 tx=0x7fa6a8011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:08.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6a8003a40 con 0x7fa6b4106da0 2023-08-25T02:19:08.509 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6a8003ba0 con 0x7fa6b4106da0 2023-08-25T02:19:08.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b419a700 con 0x7fa6b4106da0 2023-08-25T02:19:08.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6a80196c0 con 0x7fa6b4106da0 2023-08-25T02:19:08.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.967+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b419ab20 con 0x7fa6b4106da0 2023-08-25T02:19:08.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.968+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 45148+0+0 (secure 0 0 0) 0x7fa6a8019980 con 0x7fa6b4106da0 2023-08-25T02:19:08.510 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.968+0000 7fa6b0ff9700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 0x7fa69c03a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:08.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.968+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b4061f50 con 0x7fa6b4106da0 2023-08-25T02:19:08.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.968+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6a804c260 con 0x7fa6b4106da0 2023-08-25T02:19:08.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.969+0000 7fa6b2ffd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 0x7fa69c03a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:08.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.969+0000 7fa6b2ffd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 0x7fa69c03a5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006fd0 tx=0x7fa6a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:08.511 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:07.973+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa6a802c3a0 con 0x7fa6b4106da0 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.161+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa6b410a4b0 con 0x7fa6b4106da0 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.162+0000 7fa6b0ff9700 1 -- 172.21.15.89:0/701897403 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa6a8018350 con 0x7fa6b4106da0 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 msgr2=0x7fa69c03a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 0x7fa69c03a5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006fd0 tx=0x7fa6a4006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 msgr2=0x7fa6b4199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:08.512 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8006af0 tx=0x7fa6a8011a10 comp rx=0 tx=0).stop 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 shutdown_connections 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa69c038110 0x7fa69c03a5d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 --2- 172.21.15.89:0/701897403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6b4106da0 0x7fa6b4199fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 >> 172.21.15.89:0/701897403 conn(0x7fa6b4100770 msgr2=0x7fa6b4101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 shutdown_connections 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.164+0000 7fa6b9cbd700 1 -- 172.21.15.89:0/701897403 wait complete. 2023-08-25T02:19:08.513 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-08-25T02:19:08.514 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2023-08-25T02:19:08.514 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-08-25T02:19:08.514 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2023-08-25T02:19:08.514 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2023-08-25T02:19:08.514 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-08-25T02:19:08.515 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph Dashboard is now available at: 2023-08-25T02:19:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: URL: https://smithi089.front.sepia.ceph.com:8443/ 2023-08-25T02:19:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: User: admin 2023-08-25T02:19:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: Password: zxaxcm5r0h 2023-08-25T02:19:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:08.519 INFO:teuthology.orchestra.run.smithi089.stdout:Saving cluster configuration to /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config directory 2023-08-25T02:19:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling autotune for osd_memory_target 2023-08-25T02:19:09.129 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/701897403' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-08-25T02:19:09.491 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7ff72700 1 Processor -- start 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7ff72700 1 -- start start 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7ff72700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7ff72700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78105050 con 0x7f4f78104660 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7dd0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7dd0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45674/0 (socket says 172.21.15.89:45674) 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.911+0000 7f4f7dd0e700 1 -- 172.21.15.89:0/4059211411 learned_addr learned my addr 172.21.15.89:0/4059211411 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7dd0e700 1 -- 172.21.15.89:0/4059211411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f78107600 con 0x7f4f78104660 2023-08-25T02:19:09.508 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7dd0e700 1 --2- 172.21.15.89:0/4059211411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f740098a0 tx=0x7f4f74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a76e1139ba250b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:09.515 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7cd0c700 1 -- 172.21.15.89:0/4059211411 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f74004160 con 0x7f4f78104660 2023-08-25T02:19:09.516 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7cd0c700 1 -- 172.21.15.89:0/4059211411 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f740042c0 con 0x7f4f78104660 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7cd0c700 1 -- 172.21.15.89:0/4059211411 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f740119b0 con 0x7f4f78104660 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7ff72700 1 -- 172.21.15.89:0/4059211411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 msgr2=0x7f4f78104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/4059211411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f740098a0 tx=0x7f4f74009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7ff72700 1 -- 172.21.15.89:0/4059211411 shutdown_connections 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/4059211411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78104a80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:09.517 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.912+0000 7f4f7ff72700 1 -- 172.21.15.89:0/4059211411 >> 172.21.15.89:0/4059211411 conn(0x7f4f780fff00 msgr2=0x7f4f78102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:09.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 -- 172.21.15.89:0/4059211411 shutdown_connections 2023-08-25T02:19:09.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 -- 172.21.15.89:0/4059211411 wait complete. 2023-08-25T02:19:09.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 Processor -- start 2023-08-25T02:19:09.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 -- start start 2023-08-25T02:19:09.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:09.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7ff72700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7819a4f0 con 0x7f4f78104660 2023-08-25T02:19:09.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7dd0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:09.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7dd0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45676/0 (socket says 172.21.15.89:45676) 2023-08-25T02:19:09.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.913+0000 7f4f7dd0e700 1 -- 172.21.15.89:0/3816450371 learned_addr learned my addr 172.21.15.89:0/3816450371 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:09.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f7dd0e700 1 -- 172.21.15.89:0/3816450371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74009550 con 0x7f4f78104660 2023-08-25T02:19:09.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f7dd0e700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006af0 tx=0x7f4f74011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:09.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f74003a40 con 0x7f4f78104660 2023-08-25T02:19:09.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f74003ba0 con 0x7f4f78104660 2023-08-25T02:19:09.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f740196c0 con 0x7f4f78104660 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7819a6f0 con 0x7f4f78104660 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.914+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7819ab10 con 0x7f4f78104660 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 45148+0+0 (secure 0 0 0) 0x7f4f74019980 con 0x7f4f78104660 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f6effd700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 0x7f4f6403a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f7404c3b0 con 0x7f4f78104660 2023-08-25T02:19:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f78061f50 con 0x7f4f78104660 2023-08-25T02:19:09.554 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f7d50d700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 0x7f4f6403a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:09.555 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.915+0000 7f4f7d50d700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 0x7f4f6403a5d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f68006fd0 tx=0x7f4f68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:09.555 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:08.920+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4f740243f0 con 0x7f4f78104660 2023-08-25T02:19:09.555 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.106+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4f7810a800 con 0x7f4f78104660 2023-08-25T02:19:09.556 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.107+0000 7f4f6effd700 1 -- 172.21.15.89:0/3816450371 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4f74049020 con 0x7f4f78104660 2023-08-25T02:19:09.556 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 msgr2=0x7f4f6403a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:09.559 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 0x7f4f6403a5d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f68006fd0 tx=0x7f4f68006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 msgr2=0x7f4f78199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006af0 tx=0x7f4f74011a10 comp rx=0 tx=0).stop 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 shutdown_connections 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4f64038110 0x7f4f6403a5d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 --2- 172.21.15.89:0/3816450371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f78104660 0x7f4f78199fb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:09.560 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 >> 172.21.15.89:0/3816450371 conn(0x7f4f780fff00 msgr2=0x7f4f78100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:09.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 shutdown_connections 2023-08-25T02:19:09.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.109+0000 7f4f7ff72700 1 -- 172.21.15.89:0/3816450371 wait complete. 2023-08-25T02:19:10.488 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.867+0000 7f3c98da0700 1 Processor -- start 2023-08-25T02:19:10.489 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.867+0000 7f3c98da0700 1 -- start start 2023-08-25T02:19:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.867+0000 7f3c98da0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.867+0000 7f3c98da0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94074120 con 0x7f3c94106da0 2023-08-25T02:19:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45680/0 (socket says 172.21.15.89:45680) 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9259c700 1 -- 172.21.15.89:0/3573069676 learned_addr learned my addr 172.21.15.89:0/3573069676 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9259c700 1 -- 172.21.15.89:0/3573069676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c941096d0 con 0x7f3c94106da0 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9259c700 1 --2- 172.21.15.89:0/3573069676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c880098a0 tx=0x7f3c88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b42e2691b32d335 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9159a700 1 -- 172.21.15.89:0/3573069676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c88004160 con 0x7f3c94106da0 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9159a700 1 -- 172.21.15.89:0/3573069676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c880042c0 con 0x7f3c94106da0 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.868+0000 7f3c9159a700 1 -- 172.21.15.89:0/3573069676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c880119b0 con 0x7f3c94106da0 2023-08-25T02:19:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 -- 172.21.15.89:0/3573069676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 msgr2=0x7f3c94109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:10.493 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3573069676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c880098a0 tx=0x7f3c88009bb0 comp rx=0 tx=0).stop 2023-08-25T02:19:10.493 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 -- 172.21.15.89:0/3573069676 shutdown_connections 2023-08-25T02:19:10.493 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3573069676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94109190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:10.493 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 -- 172.21.15.89:0/3573069676 >> 172.21.15.89:0/3573069676 conn(0x7f3c94100770 msgr2=0x7f3c94102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 -- 172.21.15.89:0/3573069676 shutdown_connections 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.869+0000 7f3c98da0700 1 -- 172.21.15.89:0/3573069676 wait complete. 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 Processor -- start 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 -- start start 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:10.499 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9419a500 con 0x7f3c94106da0 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c9259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c9259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45682/0 (socket says 172.21.15.89:45682) 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c9259c700 1 -- 172.21.15.89:0/3633583392 learned_addr learned my addr 172.21.15.89:0/3633583392 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c9259c700 1 -- 172.21.15.89:0/3633583392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88009550 con 0x7f3c94106da0 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c9259c700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c88006af0 tx=0x7f3c88011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:10.500 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c88003a40 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9419a700 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c88003ba0 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c880196c0 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.870+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9419ab20 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.871+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 45148+0+0 (secure 0 0 0) 0x7f3c88019980 con 0x7f3c94106da0 2023-08-25T02:19:10.501 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.871+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c94061f50 con 0x7f3c94106da0 2023-08-25T02:19:10.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.871+0000 7f3c837fe700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 0x7f3c7c03a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:10.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.872+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c8804c520 con 0x7f3c94106da0 2023-08-25T02:19:10.503 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.872+0000 7f3c91d9b700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 0x7f3c7c03a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.872+0000 7f3c91d9b700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 0x7f3c7c03a5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c84006fd0 tx=0x7f3c84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.876+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f3c88018760 con 0x7f3c94106da0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:09.879+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f3c8802dcd0 con 0x7f3c94106da0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.169+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f3c9419d560 con 0x7f3c94106da0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.172+0000 7f3c837fe700 1 -- 172.21.15.89:0/3633583392 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f3c88018760 con 0x7f3c94106da0 2023-08-25T02:19:10.504 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 msgr2=0x7f3c7c03a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 0x7f3c7c03a5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c84006fd0 tx=0x7f3c84006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 msgr2=0x7f3c94199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c88006af0 tx=0x7f3c88011a10 comp rx=0 tx=0).stop 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 shutdown_connections 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3c7c038110 0x7f3c7c03a5d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 --2- 172.21.15.89:0/3633583392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c94106da0 0x7f3c94199fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:10.505 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 >> 172.21.15.89:0/3633583392 conn(0x7f3c94100770 msgr2=0x7f3c94101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 shutdown_connections 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2023-08-25T02:19:10.174+0000 7f3c98da0700 1 -- 172.21.15.89:0/3633583392 wait complete. 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.506 INFO:teuthology.orchestra.run.smithi089.stdout:Or, if you are only running a single cluster on this host: 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout:Please consider enabling telemetry to help improve Ceph: 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: ceph telemetry on 2023-08-25T02:19:10.507 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.508 INFO:teuthology.orchestra.run.smithi089.stdout:For more information see: 2023-08-25T02:19:10.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.508 INFO:teuthology.orchestra.run.smithi089.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-08-25T02:19:10.509 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:10.509 INFO:teuthology.orchestra.run.smithi089.stdout:Bootstrap complete. 2023-08-25T02:19:10.519 INFO:tasks.cephadm:Fetching config... 2023-08-25T02:19:10.520 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:10.520 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-08-25T02:19:10.537 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-08-25T02:19:10.538 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:10.538 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-08-25T02:19:10.596 INFO:tasks.cephadm:Fetching mon keyring... 2023-08-25T02:19:10.598 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:10.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-08-25T02:19:10.671 INFO:tasks.cephadm:Fetching pub ssh key... 2023-08-25T02:19:10.671 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:10.672 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-08-25T02:19:10.727 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-08-25T02:19:10.727 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-08-25T02:19:10.805 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:10.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-08-25T02:19:10.862 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:10.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-08-25T02:19:10.923 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC092pFOdMgjoOLfXrp5bvIABoJ9BxWIqkR9zvWbhRT4UIb/NckgOqyN/9lMShLXD2cbOV47flnhSUIYTwssPIlEJWhBQdImL9r8W5e+kjmRk84Pzst0O03AEcuUX8HEOMnhV6Q9BbI36yO/kA2aymSiaMqLV1pW7DyffpQTqSNHsG3aZAhv7Nt+k8WJ9H5/B8RuvGiraYNoYYeaznNa6Nz28RbZsbl/qvDXbwGIBJjPHGcz8HHnQvVLVT90NTA0Qx5y46RkIJshxtRf91Kh4Cs7SE1Czm3abKGCmGlRI0jmZeHjE8VHqE3fYPSOmsAuI1hwqSIN+8kiAgAHptdjjDOV5f6dbAAmLFSTpie+HvVJk6cD8ykYr1wH+2CQUP5lvsUyrYkVo2ITuWwGbNiayyOVznvG1/2XhZPGdGI3DWCW4b+dnJapJWTMQ7Krxse0zSjmK7+raHIXpQ1nMGiSa4R4XdiK726dLGU2IrdpBLAJHPfcaL+rAi5iFSLsG6SUS0= ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:10.937 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-08-25T02:19:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:10 smithi089 ceph-mon[98732]: mgrmap e13: x(active, since 6s) 2023-08-25T02:19:11.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3633583392' entity='client.admin' 2023-08-25T02:19:11.397 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:13.085 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:13.085 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:13.086 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:19:13.086 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:13.094 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:13.095 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:13.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2158572380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 msgr2=0x7fe0ec0ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:13.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2158572380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec0ffc60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009910 tx=0x7fe0e0009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2158572380 shutdown_connections 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2158572380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec0ffc60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2158572380 >> 172.21.15.89:0/2158572380 conn(0x7fe0ec0fa020 msgr2=0x7fe0ec0fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2158572380 shutdown_connections 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.167+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2158572380 wait complete. 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0f1fe3700 1 Processor -- start 2023-08-25T02:19:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0f1fe3700 1 -- start start 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0f1fe3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0f1fe3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec194970 con 0x7fe0ec0ff840 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0eb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0eb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45688/0 (socket says 172.21.15.89:45688) 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0eb7fe700 1 -- 172.21.15.89:0/2919592396 learned_addr learned my addr 172.21.15.89:0/2919592396 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0eb7fe700 1 -- 172.21.15.89:0/2919592396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00095f0 con 0x7fe0ec0ff840 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.168+0000 7fe0eb7fe700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0004f40 tx=0x7fe0e00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.169+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e001d070 con 0x7fe0ec0ff840 2023-08-25T02:19:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.169+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec194b70 con 0x7fe0ec0ff840 2023-08-25T02:19:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.169+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0e000bc10 con 0x7fe0ec0ff840 2023-08-25T02:19:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.169+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e0003a70 con 0x7fe0ec0ff840 2023-08-25T02:19:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.169+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec195010 con 0x7fe0ec0ff840 2023-08-25T02:19:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7fe0e0003d30 con 0x7fe0ec0ff840 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec18de30 con 0x7fe0ec0ff840 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0e8ff9700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 0x7fe0d403a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe0e004bd80 con 0x7fe0ec0ff840 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0eaffd700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 0x7fe0d403a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.170+0000 7fe0eaffd700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 0x7fe0d403a670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006fd0 tx=0x7fe0dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:13.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.176+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fe0e002abf0 con 0x7fe0ec0ff840 2023-08-25T02:19:13.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.366+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe0ec105070 con 0x7fe0ec0ff840 2023-08-25T02:19:13.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.370+0000 7fe0e8ff9700 1 -- 172.21.15.89:0/2919592396 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe0e002abf0 con 0x7fe0ec0ff840 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 msgr2=0x7fe0d403a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 0x7fe0d403a670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006fd0 tx=0x7fe0dc006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 msgr2=0x7fe0ec194430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0004f40 tx=0x7fe0e00057c0 comp rx=0 tx=0).stop 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 shutdown_connections 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe0d40381b0 0x7fe0d403a670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:13.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 --2- 172.21.15.89:0/2919592396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0ec0ff840 0x7fe0ec194430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:13.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 >> 172.21.15.89:0/2919592396 conn(0x7fe0ec0fa020 msgr2=0x7fe0ec0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 shutdown_connections 2023-08-25T02:19:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:13.372+0000 7fe0f1fe3700 1 -- 172.21.15.89:0/2919592396 wait complete. 2023-08-25T02:19:13.750 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-08-25T02:19:13.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-08-25T02:19:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:14.692 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2919592396' entity='client.admin' 2023-08-25T02:19:14.693 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:14.693 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:15.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 -- 172.21.15.89:0/899628092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 msgr2=0x7f79480ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:15.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 --2- 172.21.15.89:0/899628092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480ff730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7938009910 tx=0x7f7938009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:15.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 -- 172.21.15.89:0/899628092 shutdown_connections 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 --2- 172.21.15.89:0/899628092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480ff730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 -- 172.21.15.89:0/899628092 >> 172.21.15.89:0/899628092 conn(0x7f79480fa8b0 msgr2=0x7f79480fccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 -- 172.21.15.89:0/899628092 shutdown_connections 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.451+0000 7f794e007700 1 -- 172.21.15.89:0/899628092 wait complete. 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f794e007700 1 Processor -- start 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f794e007700 1 -- start start 2023-08-25T02:19:15.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f794e007700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f794e007700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948198250 con 0x7f79480ff310 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f79477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f79477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45694/0 (socket says 172.21.15.89:45694) 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.452+0000 7f79477fe700 1 -- 172.21.15.89:0/2312018252 learned_addr learned my addr 172.21.15.89:0/2312018252 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f79477fe700 1 -- 172.21.15.89:0/2312018252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380095f0 con 0x7f79480ff310 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f79477fe700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7938004f40 tx=0x7f79380057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f793801d070 con 0x7f79480ff310 2023-08-25T02:19:15.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f793800bc10 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7948198390 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7938003a70 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.453+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7948198830 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.454+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794818de20 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.454+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f7938003ce0 con 0x7f79480ff310 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.454+0000 7f7944ff9700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 0x7f793003a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:15.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.454+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f793804bd30 con 0x7f79480ff310 2023-08-25T02:19:15.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.454+0000 7f7946ffd700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 0x7f793003a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:15.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.455+0000 7f7946ffd700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 0x7f793003a6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f793c006fd0 tx=0x7f793c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:15.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.459+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f793802abf0 con 0x7f79480ff310 2023-08-25T02:19:15.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.652+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f794802cec0 con 0x7f7930038200 2023-08-25T02:19:15.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.655+0000 7f7944ff9700 1 -- 172.21.15.89:0/2312018252 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f794802cec0 con 0x7f7930038200 2023-08-25T02:19:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 msgr2=0x7f793003a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 0x7f793003a6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f793c006fd0 tx=0x7f793c006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 msgr2=0x7f79480686a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7938004f40 tx=0x7f79380057c0 comp rx=0 tx=0).stop 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 shutdown_connections 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7930038200 0x7f793003a6c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 --2- 172.21.15.89:0/2312018252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79480ff310 0x7f79480686a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:15.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 >> 172.21.15.89:0/2312018252 conn(0x7f79480fa8b0 msgr2=0x7f79480fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:15.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 shutdown_connections 2023-08-25T02:19:15.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:15.657+0000 7f794e007700 1 -- 172.21.15.89:0/2312018252 wait complete. 2023-08-25T02:19:16.057 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi121 2023-08-25T02:19:16.058 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:19:16.059 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.conf 2023-08-25T02:19:16.082 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:19:16.082 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:16.143 INFO:tasks.cephadm:Adding host smithi121 to orchestrator... 2023-08-25T02:19:16.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch host add smithi121 2023-08-25T02:19:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:16.928 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:16.928 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:16.928 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:19:16.929 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:16.930 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:16.930 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 -- 172.21.15.89:0/1013317419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 msgr2=0x7f1d50105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 --2- 172.21.15.89:0/1013317419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50105710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009910 tx=0x7f1d40009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 -- 172.21.15.89:0/1013317419 shutdown_connections 2023-08-25T02:19:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 --2- 172.21.15.89:0/1013317419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50105710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 -- 172.21.15.89:0/1013317419 >> 172.21.15.89:0/1013317419 conn(0x7f1d50074e20 msgr2=0x7f1d50075230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 -- 172.21.15.89:0/1013317419 shutdown_connections 2023-08-25T02:19:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.865+0000 7f1d57526700 1 -- 172.21.15.89:0/1013317419 wait complete. 2023-08-25T02:19:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d57526700 1 Processor -- start 2023-08-25T02:19:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d57526700 1 -- start start 2023-08-25T02:19:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d57526700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d57526700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d50196bd0 con 0x7f1d50105330 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d552c2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d552c2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45700/0 (socket says 172.21.15.89:45700) 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d552c2700 1 -- 172.21.15.89:0/2916548751 learned_addr learned my addr 172.21.15.89:0/2916548751 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d552c2700 1 -- 172.21.15.89:0/2916548751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400095f0 con 0x7f1d50105330 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d552c2700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d40004f40 tx=0x7f1d40005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d4001d070 con 0x7f1d50105330 2023-08-25T02:19:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d40004370 con 0x7f1d50105330 2023-08-25T02:19:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d40003970 con 0x7f1d50105330 2023-08-25T02:19:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.866+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d50196e30 con 0x7f1d50105330 2023-08-25T02:19:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.867+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d501972b0 con 0x7f1d50105330 2023-08-25T02:19:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.867+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f1d40003be0 con 0x7f1d50105330 2023-08-25T02:19:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.867+0000 7f1d467fc700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 0x7f1d3c03a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.867+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1d4004be90 con 0x7f1d50105330 2023-08-25T02:19:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.868+0000 7f1d54ac1700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 0x7f1d3c03a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.868+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d501974f0 con 0x7f1d50105330 2023-08-25T02:19:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.868+0000 7f1d54ac1700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 0x7f1d3c03a6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c006fd0 tx=0x7f1d4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:17.872+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1d40026070 con 0x7f1d50105330 2023-08-25T02:19:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:18.064+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d50068130 con 0x7f1d3c038200 2023-08-25T02:19:19.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:18 smithi089 ceph-mon[98732]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:19.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:18 smithi089 ceph-mon[98732]: Deploying cephadm binary to smithi121 2023-08-25T02:19:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.377+0000 7f1d467fc700 1 -- 172.21.15.89:0/2916548751 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1d50068130 con 0x7f1d3c038200 2023-08-25T02:19:20.379 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi121' with addr '172.21.15.121' 2023-08-25T02:19:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 msgr2=0x7f1d3c03a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 0x7f1d3c03a6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c006fd0 tx=0x7f1d4c006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 msgr2=0x7f1d50196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d40004f40 tx=0x7f1d40005020 comp rx=0 tx=0).stop 2023-08-25T02:19:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 shutdown_connections 2023-08-25T02:19:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d3c038200 0x7f1d3c03a6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 --2- 172.21.15.89:0/2916548751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d50105330 0x7f1d50196690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 >> 172.21.15.89:0/2916548751 conn(0x7f1d50074e20 msgr2=0x7f1d500fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.379+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 shutdown_connections 2023-08-25T02:19:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:20.380+0000 7f1d57526700 1 -- 172.21.15.89:0/2916548751 wait complete. 2023-08-25T02:19:20.766 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch host ls --format=json 2023-08-25T02:19:21.320 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:21.676 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:21 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:21.676 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:21 smithi089 ceph-mon[98732]: Added host smithi121 2023-08-25T02:19:21.676 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:21 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:21.677 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:21 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 -- 172.21.15.89:0/860355928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 msgr2=0x7f54d81009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:22.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 --2- 172.21.15.89:0/860355928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d81009d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009910 tx=0x7f54c8009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 -- 172.21.15.89:0/860355928 shutdown_connections 2023-08-25T02:19:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 --2- 172.21.15.89:0/860355928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d81009d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 -- 172.21.15.89:0/860355928 >> 172.21.15.89:0/860355928 conn(0x7f54d80fbe80 msgr2=0x7f54d80fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 -- 172.21.15.89:0/860355928 shutdown_connections 2023-08-25T02:19:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.613+0000 7f54de858700 1 -- 172.21.15.89:0/860355928 wait complete. 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.614+0000 7f54de858700 1 Processor -- start 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.614+0000 7f54de858700 1 -- start start 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54de858700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54de858700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d819bb60 con 0x7f54d81005f0 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45726/0 (socket says 172.21.15.89:45726) 2023-08-25T02:19:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d7fff700 1 -- 172.21.15.89:0/3056387823 learned_addr learned my addr 172.21.15.89:0/3056387823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d7fff700 1 -- 172.21.15.89:0/3056387823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80095f0 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d7fff700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f54c8004f40 tx=0x7f54c8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54c801d070 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54c8004370 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d819bd60 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54c80039b0 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.615+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d819c180 con 0x7f54d81005f0 2023-08-25T02:19:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.616+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f54c8003c70 con 0x7f54d81005f0 2023-08-25T02:19:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.616+0000 7f54d57fa700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 0x7f54c403a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.616+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f54c804cca0 con 0x7f54d81005f0 2023-08-25T02:19:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.616+0000 7f54d77fe700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 0x7f54c403a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.616+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d8061f50 con 0x7f54d81005f0 2023-08-25T02:19:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.617+0000 7f54d77fe700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 0x7f54c403a670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.621+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f54c8026070 con 0x7f54d81005f0 2023-08-25T02:19:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.813+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54d810a780 con 0x7f54c40381b0 2023-08-25T02:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.814+0000 7f54d57fa700 1 -- 172.21.15.89:0/3056387823 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f54d810a780 con 0x7f54c40381b0 2023-08-25T02:19:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:22.816 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}] 2023-08-25T02:19:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.816+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 msgr2=0x7f54c403a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.816+0000 7f54de858700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 0x7f54c403a670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.816+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 msgr2=0x7f54d819b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.816+0000 7f54de858700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f54c8004f40 tx=0x7f54c8005020 comp rx=0 tx=0).stop 2023-08-25T02:19:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 shutdown_connections 2023-08-25T02:19:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54c40381b0 0x7f54c403a670 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 --2- 172.21.15.89:0/3056387823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54d81005f0 0x7f54d819b620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 >> 172.21.15.89:0/3056387823 conn(0x7f54d80fbe80 msgr2=0x7f54d8106150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 shutdown_connections 2023-08-25T02:19:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:22.817+0000 7f54de858700 1 -- 172.21.15.89:0/3056387823 wait complete. 2023-08-25T02:19:22.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:22.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:23.162 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2023-08-25T02:19:23.162 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:19:23.163 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2023-08-25T02:19:23.187 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:19:23.187 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:23.247 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2023-08-25T02:19:23.248 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch host add smithi122 2023-08-25T02:19:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:24.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:24 smithi089 ceph-mon[98732]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:19:24.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:24 smithi089 ceph-mon[98732]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 -- 172.21.15.89:0/2814954437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 msgr2=0x7f1f30105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 --2- 172.21.15.89:0/2814954437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f30105710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009910 tx=0x7f1f20009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 -- 172.21.15.89:0/2814954437 shutdown_connections 2023-08-25T02:19:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 --2- 172.21.15.89:0/2814954437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f30105710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 -- 172.21.15.89:0/2814954437 >> 172.21.15.89:0/2814954437 conn(0x7f1f30074e20 msgr2=0x7f1f30075230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 -- 172.21.15.89:0/2814954437 shutdown_connections 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.838+0000 7f1f386df700 1 -- 172.21.15.89:0/2814954437 wait complete. 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f386df700 1 Processor -- start 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f386df700 1 -- start start 2023-08-25T02:19:24.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f386df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f386df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3019aa50 con 0x7f1f30105330 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f3647b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f3647b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45732/0 (socket says 172.21.15.89:45732) 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.839+0000 7f1f3647b700 1 -- 172.21.15.89:0/95533197 learned_addr learned my addr 172.21.15.89:0/95533197 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f3647b700 1 -- 172.21.15.89:0/95533197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200095f0 con 0x7f1f30105330 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f3647b700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004f40 tx=0x7f1f20005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:24.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f2001d070 con 0x7f1f30105330 2023-08-25T02:19:24.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3019acb0 con 0x7f1f30105330 2023-08-25T02:19:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f20004370 con 0x7f1f30105330 2023-08-25T02:19:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f20003970 con 0x7f1f30105330 2023-08-25T02:19:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.840+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f30194590 con 0x7f1f30105330 2023-08-25T02:19:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f1f20003c30 con 0x7f1f30105330 2023-08-25T02:19:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f30061f50 con 0x7f1f30105330 2023-08-25T02:19:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f277fe700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 0x7f1f1c03a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1f2004bd60 con 0x7f1f30105330 2023-08-25T02:19:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f35c7a700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 0x7f1f1c03a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.841+0000 7f1f35c7a700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 0x7f1f1c03a670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c006fd0 tx=0x7f1f2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:24.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:24.846+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1f2002a590 con 0x7f1f30105330 2023-08-25T02:19:25.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:25.050+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f301095d0 con 0x7f1f1c0381b0 2023-08-25T02:19:26.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:26.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: Deploying cephadm binary to smithi122 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:26.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:27.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:27 smithi089 ceph-mon[98732]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:19:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.362+0000 7f1f277fe700 1 -- 172.21.15.89:0/95533197 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1f301095d0 con 0x7f1f1c0381b0 2023-08-25T02:19:27.371 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi122' with addr '172.21.15.122' 2023-08-25T02:19:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 msgr2=0x7f1f1c03a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 0x7f1f1c03a670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c006fd0 tx=0x7f1f2c006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 msgr2=0x7f1f3019a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004f40 tx=0x7f1f20005020 comp rx=0 tx=0).stop 2023-08-25T02:19:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 shutdown_connections 2023-08-25T02:19:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1f1c0381b0 0x7f1f1c03a670 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 --2- 172.21.15.89:0/95533197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f30105330 0x7f1f3019a510 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 >> 172.21.15.89:0/95533197 conn(0x7f1f30074e20 msgr2=0x7f1f300fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 shutdown_connections 2023-08-25T02:19:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:27.365+0000 7f1f386df700 1 -- 172.21.15.89:0/95533197 wait complete. 2023-08-25T02:19:27.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch host ls --format=json 2023-08-25T02:19:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:28.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:28 smithi089 ceph-mon[98732]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:28.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:28 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:28.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:28 smithi089 ceph-mon[98732]: Added host smithi122 2023-08-25T02:19:28.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:28 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:28.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:28 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:29.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.361+0000 7f5710fde700 1 -- 172.21.15.89:0/4292663427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 msgr2=0x7f570c106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.361+0000 7f5710fde700 1 --2- 172.21.15.89:0/4292663427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c106820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009910 tx=0x7f56fc009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.362+0000 7f5710fde700 1 -- 172.21.15.89:0/4292663427 shutdown_connections 2023-08-25T02:19:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.362+0000 7f5710fde700 1 --2- 172.21.15.89:0/4292663427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c106820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.362+0000 7f5710fde700 1 -- 172.21.15.89:0/4292663427 >> 172.21.15.89:0/4292663427 conn(0x7f570c074e20 msgr2=0x7f570c075230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.362+0000 7f5710fde700 1 -- 172.21.15.89:0/4292663427 shutdown_connections 2023-08-25T02:19:29.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.362+0000 7f5710fde700 1 -- 172.21.15.89:0/4292663427 wait complete. 2023-08-25T02:19:29.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f5710fde700 1 Processor -- start 2023-08-25T02:19:29.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f5710fde700 1 -- start start 2023-08-25T02:19:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f5710fde700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:29.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f5710fde700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c19bb50 con 0x7f570c106440 2023-08-25T02:19:29.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f570a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:29.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f570a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45758/0 (socket says 172.21.15.89:45758) 2023-08-25T02:19:29.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f570a59c700 1 -- 172.21.15.89:0/1119938398 learned_addr learned my addr 172.21.15.89:0/1119938398 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:29.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f570a59c700 1 -- 172.21.15.89:0/1119938398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0095f0 con 0x7f570c106440 2023-08-25T02:19:29.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.363+0000 7f570a59c700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56fc004f40 tx=0x7f56fc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:29.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56fc01d070 con 0x7f570c106440 2023-08-25T02:19:29.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56fc004370 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c19bdb0 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56fc003970 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c195690 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f56fc003be0 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.364+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c061f50 con 0x7f570c106440 2023-08-25T02:19:29.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.365+0000 7f56fb7fe700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 0x7f56f403a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:29.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.365+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56fc04bce0 con 0x7f570c106440 2023-08-25T02:19:29.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.365+0000 7f5709d9b700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 0x7f56f403a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:29.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.365+0000 7f5709d9b700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 0x7f56f403a6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5700006fd0 tx=0x7f5700006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:29.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.369+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f56fc029950 con 0x7f570c106440 2023-08-25T02:19:29.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.565+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f570c10a710 con 0x7f56f4038200 2023-08-25T02:19:29.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.566+0000 7f56fb7fe700 1 -- 172.21.15.89:0/1119938398 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f570c10a710 con 0x7f56f4038200 2023-08-25T02:19:29.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:19:29.569 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2023-08-25T02:19:29.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 msgr2=0x7f56f403a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:29.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 0x7f56f403a6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5700006fd0 tx=0x7f5700006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:29.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 msgr2=0x7f570c19b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:29.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56fc004f40 tx=0x7f56fc005020 comp rx=0 tx=0).stop 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 shutdown_connections 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f56f4038200 0x7f56f403a6c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 --2- 172.21.15.89:0/1119938398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c106440 0x7f570c19b610 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 >> 172.21.15.89:0/1119938398 conn(0x7f570c074e20 msgr2=0x7f570c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 shutdown_connections 2023-08-25T02:19:29.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:29.568+0000 7f5710fde700 1 -- 172.21.15.89:0/1119938398 wait complete. 2023-08-25T02:19:29.697 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:29.978 INFO:tasks.cephadm:Setting crush tunables to default 2023-08-25T02:19:29.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd crush tunables default 2023-08-25T02:19:30.553 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:30.773 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:30 smithi089 ceph-mon[98732]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:30.774 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:30 smithi089 ceph-mon[98732]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:19:30.774 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:30 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 -- 172.21.15.89:0/1664467663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 msgr2=0x7f0f541009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/1664467663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f541009f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009910 tx=0x7f0f48009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 -- 172.21.15.89:0/1664467663 shutdown_connections 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/1664467663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f541009f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 -- 172.21.15.89:0/1664467663 >> 172.21.15.89:0/1664467663 conn(0x7f0f540fbee0 msgr2=0x7f0f540fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 -- 172.21.15.89:0/1664467663 shutdown_connections 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.629+0000 7f0f5a03e700 1 -- 172.21.15.89:0/1664467663 wait complete. 2023-08-25T02:19:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f5a03e700 1 Processor -- start 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f5a03e700 1 -- start start 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f5a03e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f5a03e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419bb60 con 0x7f0f54100610 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45764/0 (socket says 172.21.15.89:45764) 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f537fe700 1 -- 172.21.15.89:0/442069125 learned_addr learned my addr 172.21.15.89:0/442069125 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.630+0000 7f0f537fe700 1 -- 172.21.15.89:0/442069125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f480095f0 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f537fe700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0f48004f40 tx=0x7f0f48005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f4801d070 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f48004370 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f480039b0 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5419bd60 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.631+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5419c180 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f0f48003c20 con 0x7f0f54100610 2023-08-25T02:19:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f54061f50 con 0x7f0f54100610 2023-08-25T02:19:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f50ff9700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 0x7f0f3c03a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0f4804bfa0 con 0x7f0f54100610 2023-08-25T02:19:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f52ffd700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 0x7f0f3c03a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.632+0000 7f0f52ffd700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 0x7f0f3c03a6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f44006fd0 tx=0x7f0f44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:31.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.637+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0f48029950 con 0x7f0f54100610 2023-08-25T02:19:31.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:31.831+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0f54196bb0 con 0x7f0f54100610 2023-08-25T02:19:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.445+0000 7f0f50ff9700 1 -- 172.21.15.89:0/442069125 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0f48026070 con 0x7f0f54100610 2023-08-25T02:19:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 msgr2=0x7f0f3c03a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 0x7f0f3c03a6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f44006fd0 tx=0x7f0f44006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 msgr2=0x7f0f5419b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0f48004f40 tx=0x7f0f48005020 comp rx=0 tx=0).stop 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 shutdown_connections 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0f3c038200 0x7f0f3c03a6c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 --2- 172.21.15.89:0/442069125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f54100610 0x7f0f5419b620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 >> 172.21.15.89:0/442069125 conn(0x7f0f540fbee0 msgr2=0x7f0f54106170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 shutdown_connections 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:32.447+0000 7f0f5a03e700 1 -- 172.21.15.89:0/442069125 wait complete. 2023-08-25T02:19:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2023-08-25T02:19:32.779 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:32.779 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:32.780 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:32.819 INFO:tasks.cephadm:Adding mon.a on smithi089 2023-08-25T02:19:32.819 INFO:tasks.cephadm:Adding mon.b on smithi121 2023-08-25T02:19:32.819 INFO:tasks.cephadm:Adding mon.c on smithi122 2023-08-25T02:19:32.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch apply mon '3;smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c' 2023-08-25T02:19:33.410 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-08-25T02:19:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:34.641 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 -- 172.21.15.122:0/1455033580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 msgr2=0x7f45801009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:34.641 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1455033580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f45801009d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4570009910 tx=0x7f4570009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:34.641 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 -- 172.21.15.122:0/1455033580 shutdown_connections 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1455033580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f45801009d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 -- 172.21.15.122:0/1455033580 >> 172.21.15.122:0/1455033580 conn(0x7f45800fbe80 msgr2=0x7f45800fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.639+0000 7f4587a2c700 1 -- 172.21.15.122:0/1455033580 shutdown_connections 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.640+0000 7f4587a2c700 1 -- 172.21.15.122:0/1455033580 wait complete. 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.640+0000 7f4587a2c700 1 Processor -- start 2023-08-25T02:19:34.642 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.640+0000 7f4587a2c700 1 -- start start 2023-08-25T02:19:34.643 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f4587a2c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:34.643 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f4587a2c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458019bb60 con 0x7f45801005f0 2023-08-25T02:19:34.645 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45857c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:34.645 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45857c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.122:48252/0 (socket says 172.21.15.122:48252) 2023-08-25T02:19:34.645 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45857c8700 1 -- 172.21.15.122:0/1093674794 learned_addr learned my addr 172.21.15.122:0/1093674794 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:19:34.646 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45857c8700 1 -- 172.21.15.122:0/1093674794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700095f0 con 0x7f45801005f0 2023-08-25T02:19:34.646 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45857c8700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4570004f40 tx=0x7f4570005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:34.646 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f457001d070 con 0x7f45801005f0 2023-08-25T02:19:34.655 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458019bd60 con 0x7f45801005f0 2023-08-25T02:19:34.655 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4570004370 con 0x7f45801005f0 2023-08-25T02:19:34.655 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45700039b0 con 0x7f45801005f0 2023-08-25T02:19:34.656 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.641+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458019c180 con 0x7f45801005f0 2023-08-25T02:19:34.656 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.642+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f4570003c70 con 0x7f45801005f0 2023-08-25T02:19:34.656 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.642+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4580061f50 con 0x7f45801005f0 2023-08-25T02:19:34.656 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.643+0000 7f45767fc700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 0x7f456c03a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:34.656 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.643+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f457004bff0 con 0x7f45801005f0 2023-08-25T02:19:34.657 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.643+0000 7f4584fc7700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 0x7f456c03a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:34.657 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.643+0000 7f4584fc7700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 0x7f456c03a670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:34.659 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.648+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4570029360 con 0x7f45801005f0 2023-08-25T02:19:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:34 smithi089 ceph-mon[98732]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:34 smithi089 ceph-mon[98732]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:19:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:34 smithi089 ceph-mon[98732]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:34.848 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.847+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f4580000990 con 0x7f456c0381b0 2023-08-25T02:19:34.855 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.853+0000 7f45767fc700 1 -- 172.21.15.122:0/1093674794 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4580000990 con 0x7f456c0381b0 2023-08-25T02:19:34.855 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2023-08-25T02:19:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 msgr2=0x7f456c03a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 0x7f456c03a670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:34.858 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 msgr2=0x7f458019b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:34.859 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4570004f40 tx=0x7f4570005020 comp rx=0 tx=0).stop 2023-08-25T02:19:34.862 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 shutdown_connections 2023-08-25T02:19:34.862 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f456c0381b0 0x7f456c03a670 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:34.862 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 --2- 172.21.15.122:0/1093674794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45801005f0 0x7f458019b620 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:34.863 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 >> 172.21.15.122:0/1093674794 conn(0x7f45800fbe80 msgr2=0x7f4580106150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:34.863 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.855+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 shutdown_connections 2023-08-25T02:19:34.875 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:34.856+0000 7f4587a2c700 1 -- 172.21.15.122:0/1093674794 wait complete. 2023-08-25T02:19:35.310 DEBUG:teuthology.orchestra.run.smithi121:mon.b> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.b.service 2023-08-25T02:19:35.321 DEBUG:teuthology.orchestra.run.smithi122:mon.c> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.c.service 2023-08-25T02:19:35.327 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-08-25T02:19:35.365 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph mon dump -f json 2023-08-25T02:19:35.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:19:35.368 INFO:journalctl@ceph.mon.c.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:19:36.192 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:36.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c;count:3 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: Deploying daemon mon.c on smithi122 2023-08-25T02:19:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:35 smithi089 ceph-mon[98732]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:37.869 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.867+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1713642329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 msgr2=0x7f0bc0104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:37.869 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.867+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1713642329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc0104e00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009910 tx=0x7f0bbc009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1713642329 shutdown_connections 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1713642329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc0104e00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1713642329 >> 172.21.15.122:0/1713642329 conn(0x7f0bc0075980 msgr2=0x7f0bc0075d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1713642329 shutdown_connections 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1713642329 wait complete. 2023-08-25T02:19:37.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 Processor -- start 2023-08-25T02:19:37.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.868+0000 7f0bc7b85700 1 -- start start 2023-08-25T02:19:37.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc7b85700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:37.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc7b85700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc019aa50 con 0x7f0bc0104a20 2023-08-25T02:19:37.921 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc5921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc5921700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.122:48258/0 (socket says 172.21.15.122:48258) 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc5921700 1 -- 172.21.15.122:0/1331185362 learned_addr learned my addr 172.21.15.122:0/1331185362 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc5921700 1 -- 172.21.15.122:0/1331185362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0095f0 con 0x7f0bc0104a20 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.869+0000 7f0bc5921700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc004f40 tx=0x7f0bbc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.870+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bbc01d070 con 0x7f0bc0104a20 2023-08-25T02:19:37.922 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.870+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bbc004370 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.870+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc019acb0 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.870+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bbc003970 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.870+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc0194590 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f0bbc003c30 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0061f50 con 0x7f0bc0104a20 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bb6ffd700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 0x7f0bac03a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:37.923 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0bbc04bd60 con 0x7f0bc0104a20 2023-08-25T02:19:37.924 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bc5120700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 0x7f0bac03a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:37.924 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.871+0000 7f0bc5120700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 0x7f0bac03a670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006fd0 tx=0x7f0bb0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:37.924 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:37.876+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0bbc02a590 con 0x7f0bc0104a20 2023-08-25T02:19:38.180 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.177+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bc0195aa0 con 0x7f0bc0104a20 2023-08-25T02:19:38.181 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.178+0000 7f0bb6ffd700 1 -- 172.21.15.122:0/1331185362 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f0bbc026030 con 0x7f0bc0104a20 2023-08-25T02:19:38.187 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:19:38.187 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","modified":"2023-08-25T02:18:06.484229Z","created":"2023-08-25T02:18:06.484229Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-08-25T02:19:38.188 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 msgr2=0x7f0bac03a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:38.188 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 0x7f0bac03a670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006fd0 tx=0x7f0bb0006e40 comp rx=0 tx=0).stop 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 msgr2=0x7f0bc019a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc004f40 tx=0x7f0bbc005020 comp rx=0 tx=0).stop 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 shutdown_connections 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0bac0381b0 0x7f0bac03a670 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 --2- 172.21.15.122:0/1331185362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc0104a20 0x7f0bc019a510 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 >> 172.21.15.122:0/1331185362 conn(0x7f0bc0075980 msgr2=0x7f0bc00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:38.189 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 shutdown_connections 2023-08-25T02:19:38.190 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:38.180+0000 7f0bc7b85700 1 -- 172.21.15.122:0/1331185362 wait complete. 2023-08-25T02:19:38.190 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2023-08-25T02:19:38.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:38 smithi089 ceph-mon[98732]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:38.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:38 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/1331185362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:38.916 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 podman[100396]: 2023-08-25T02:19:39.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: set uid:gid to 167:167 (ceph:ceph) 2023-08-25T02:19:39.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev), process ceph-mon, pid 7 2023-08-25T02:19:39.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 bash[100396]: ab4abb6069324bd39e4abbdf60bd7a51951ea6e5cb8efe4121a9476f85d5498b 2023-08-25T02:19:39.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: pidfile_write: ignore empty --pid-file 2023-08-25T02:19:39.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: load: jerasure load: lrc 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: RocksDB version: 7.9.2 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Git sha 0 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Compile date 2023-08-24 22:08:31 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: DB SUMMARY 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: DB Session ID: Q4KOG0185VC6WHRLMUZL 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: CURRENT file: CURRENT 2023-08-25T02:19:39.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: IDENTITY file: IDENTITY 2023-08-25T02:19:39.235 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-08-25T02:19:39.236 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-08-25T02:19:39.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-08-25T02:19:39.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.error_if_exists: 0 2023-08-25T02:19:39.245 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.create_if_missing: 0 2023-08-25T02:19:39.245 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.paranoid_checks: 1 2023-08-25T02:19:39.245 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.flush_verify_memtable_count: 1 2023-08-25T02:19:39.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-08-25T02:19:39.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-08-25T02:19:39.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.env: 0x55711ebd9da0 2023-08-25T02:19:39.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.fs: PosixFileSystem 2023-08-25T02:19:39.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.info_log: 0x55711fa91440 2023-08-25T02:19:39.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_file_opening_threads: 16 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.statistics: (nil) 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.use_fsync: 0 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_log_file_size: 0 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.log_file_time_to_roll: 0 2023-08-25T02:19:39.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.keep_log_file_num: 1000 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.recycle_log_file_num: 0 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_fallocate: 1 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_mmap_reads: 0 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_mmap_writes: 0 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.use_direct_reads: 0 2023-08-25T02:19:39.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-08-25T02:19:39.252 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.create_missing_column_families: 0 2023-08-25T02:19:39.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.db_log_dir: 2023-08-25T02:19:39.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.wal_dir: 2023-08-25T02:19:39.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.table_cache_numshardbits: 6 2023-08-25T02:19:39.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.WAL_ttl_seconds: 0 2023-08-25T02:19:39.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.WAL_size_limit_MB: 0 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.is_fd_close_on_exec: 1 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.advise_random_on_open: 1 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.db_write_buffer_size: 0 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.write_buffer_manager: 0x55711fa4b4a0 2023-08-25T02:19:39.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-08-25T02:19:39.255 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-08-25T02:19:39.255 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.use_adaptive_mutex: 0 2023-08-25T02:19:39.255 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.rate_limiter: (nil) 2023-08-25T02:19:39.255 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-08-25T02:19:39.255 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.wal_recovery_mode: 2 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enable_thread_tracking: 0 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enable_pipelined_write: 0 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.unordered_write: 0 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-08-25T02:19:39.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.row_cache: None 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.wal_filter: None 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_ingest_behind: 0 2023-08-25T02:19:39.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.two_write_queues: 0 2023-08-25T02:19:39.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.manual_wal_flush: 0 2023-08-25T02:19:39.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.wal_compression: 0 2023-08-25T02:19:39.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.atomic_flush: 0 2023-08-25T02:19:39.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-08-25T02:19:39.260 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.persist_stats_to_disk: 0 2023-08-25T02:19:39.260 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.write_dbid_to_manifest: 0 2023-08-25T02:19:39.261 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.log_readahead_size: 0 2023-08-25T02:19:39.261 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-08-25T02:19:39.261 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.best_efforts_recovery: 0 2023-08-25T02:19:39.261 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-08-25T02:19:39.261 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-08-25T02:19:39.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.allow_data_in_errors: 0 2023-08-25T02:19:39.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.db_host_id: __hostname__ 2023-08-25T02:19:39.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enforce_single_del_contracts: true 2023-08-25T02:19:39.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_background_jobs: 2 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_background_compactions: -1 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_subcompactions: 1 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.delayed_write_rate : 16777216 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_total_wal_size: 0 2023-08-25T02:19:39.264 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-08-25T02:19:39.265 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.stats_dump_period_sec: 600 2023-08-25T02:19:39.265 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.stats_persist_period_sec: 600 2023-08-25T02:19:39.265 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-08-25T02:19:39.265 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_open_files: -1 2023-08-25T02:19:39.265 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bytes_per_sync: 0 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.wal_bytes_per_sync: 0 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.strict_bytes_per_sync: 0 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_readahead_size: 0 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_background_flushes: -1 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Compression algorithms supported: 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-08-25T02:19:39.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kZSTD supported: 0 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kXpressCompression supported: 0 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kLZ4HCCompression supported: 1 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kZlibCompression supported: 1 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kSnappyCompression supported: 1 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kLZ4Compression supported: 1 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: kBZip2Compression supported: 0 2023-08-25T02:19:39.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Fast CRC32 supported: Supported on x86 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: DMutex implementation: pthread_mutex_t 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.merge_operator: 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_filter: None 2023-08-25T02:19:39.268 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_filter_factory: None 2023-08-25T02:19:39.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.sst_partitioner_factory: None 2023-08-25T02:19:39.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.memtable_factory: SkipListFactory 2023-08-25T02:19:39.277 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.table_factory: BlockBasedTable 2023-08-25T02:19:39.277 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55711fa91540) 2023-08-25T02:19:39.277 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks: 1 2023-08-25T02:19:39.277 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_top_level_index_and_filter: 1 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_type: 0 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_index_type: 0 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_shortening: 1 2023-08-25T02:19:39.278 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2023-08-25T02:19:39.279 INFO:journalctl@ceph.mon.c.smithi122.stdout: checksum: 4 2023-08-25T02:19:39.279 INFO:journalctl@ceph.mon.c.smithi122.stdout: no_block_cache: 0 2023-08-25T02:19:39.281 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache: 0x55711fae91f0 2023-08-25T02:19:39.281 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_name: BinnedLRUCache 2023-08-25T02:19:39.281 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_options: 2023-08-25T02:19:39.281 INFO:journalctl@ceph.mon.c.smithi122.stdout: capacity : 536870912 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_shard_bits : 4 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: strict_capacity_limit : 0 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: high_pri_pool_ratio: 0.000 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_compressed: (nil) 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: persistent_cache: (nil) 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size: 4096 2023-08-25T02:19:39.282 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size_deviation: 10 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_restart_interval: 16 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_block_restart_interval: 1 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: metadata_block_size: 4096 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: partition_filters: 0 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: use_delta_encoding: 1 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: filter_policy: bloomfilter 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: whole_key_filtering: 1 2023-08-25T02:19:39.283 INFO:journalctl@ceph.mon.c.smithi122.stdout: verify_compression: 0 2023-08-25T02:19:39.284 INFO:journalctl@ceph.mon.c.smithi122.stdout: read_amp_bytes_per_bit: 0 2023-08-25T02:19:39.284 INFO:journalctl@ceph.mon.c.smithi122.stdout: format_version: 5 2023-08-25T02:19:39.285 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_index_compression: 1 2023-08-25T02:19:39.286 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_align: 0 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout: max_auto_readahead_size: 262144 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout: prepopulate_block_cache: 0 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout: initial_auto_readahead_size: 8192 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.write_buffer_size: 33554432 2023-08-25T02:19:39.287 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_write_buffer_number: 2 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression: NoCompression 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression: Disabled 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.prefix_extractor: nullptr 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.num_levels: 7 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-08-25T02:19:39.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:19:39.289 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.window_bits: -14 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.level: 32767 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.strategy: 0 2023-08-25T02:19:39.290 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-08-25T02:19:39.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:19:39.305 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:19:39.305 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-08-25T02:19:39.305 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.enabled: false 2023-08-25T02:19:39.305 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.target_file_size_base: 67108864 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.target_file_size_multiplier: 1 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-08-25T02:19:39.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-08-25T02:19:39.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-08-25T02:19:39.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-08-25T02:19:39.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-08-25T02:19:39.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-08-25T02:19:39.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-08-25T02:19:39.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.arena_block_size: 1048576 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.disable_auto_compactions: 0 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-08-25T02:19:39.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-08-25T02:19:39.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-08-25T02:19:39.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-08-25T02:19:39.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-08-25T02:19:39.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-08-25T02:19:39.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-08-25T02:19:39.314 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-08-25T02:19:39.314 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.table_properties_collectors: 2023-08-25T02:19:39.314 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.inplace_update_support: 0 2023-08-25T02:19:39.314 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.inplace_update_num_locks: 10000 2023-08-25T02:19:39.314 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-08-25T02:19:39.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-08-25T02:19:39.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.memtable_huge_page_size: 0 2023-08-25T02:19:39.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.bloom_locality: 0 2023-08-25T02:19:39.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.max_successive_merges: 0 2023-08-25T02:19:39.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.optimize_filters_for_hits: 0 2023-08-25T02:19:39.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.paranoid_file_checks: 0 2023-08-25T02:19:39.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.force_consistency_checks: 1 2023-08-25T02:19:39.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.report_bg_io_stats: 0 2023-08-25T02:19:39.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.ttl: 2592000 2023-08-25T02:19:39.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.periodic_compaction_seconds: 0 2023-08-25T02:19:39.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enable_blob_files: false 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.min_blob_size: 0 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_file_size: 268435456 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_compression_type: NoCompression 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.enable_blob_garbage_collection: false 2023-08-25T02:19:39.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.blob_file_starting_level: 0 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-08-25T02:19:39.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ce9b93da-e32a-487e-a07b-7b5a501e5da8 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929978917423, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929978917835, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1692929978, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ce9b93da-e32a-487e-a07b-7b5a501e5da8", "db_session_id": "Q4KOG0185VC6WHRLMUZL", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929978917900, "job": 1, "event": "recovery_finished"} 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55711fb86000 2023-08-25T02:19:39.320 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: DB pointer 0x55711fb72000 2023-08-25T02:19:39.321 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** DB Stats ** 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:19:39.323 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:19:39.324 INFO:journalctl@ceph.mon.c.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:19:39.325 INFO:journalctl@ceph.mon.c.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:19:39.326 INFO:journalctl@ceph.mon.c.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-08-25T02:19:39.327 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-08-25T02:19:39.327 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-08-25T02:19:39.327 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-08-25T02:19:39.328 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2023-08-25T02:19:39.328 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:19:39.328 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:19:39.330 INFO:journalctl@ceph.mon.c.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-08-25T02:19:39.330 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache BinnedLRUCache@0x55711fae91f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-08-25T02:19:39.330 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-08-25T02:19:39.331 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.331 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2023-08-25T02:19:39.331 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-08-25T02:19:39.331 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: starting mon.c rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(???) e0 preinit fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).mds e1 new map 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).mds e1 print_map 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout: e1 2023-08-25T02:19:39.332 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-08-25T02:19:39.333 INFO:journalctl@ceph.mon.c.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-08-25T02:19:39.333 INFO:journalctl@ceph.mon.c.smithi122.stdout: legacy client fscid: -1 2023-08-25T02:19:39.333 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2023-08-25T02:19:39.334 INFO:journalctl@ceph.mon.c.smithi122.stdout: No filesystems configured 2023-08-25T02:19:39.334 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-08-25T02:19:39.334 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-08-25T02:19:39.334 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-08-25T02:19:39.335 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-08-25T02:19:39.335 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-08-25T02:19:39.335 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-08-25T02:19:39.338 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-08-25T02:19:39.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:19:39.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:39.341 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:39.341 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-08-25T02:19:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:19:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c;count:3 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:39.344 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:39.345 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:39.351 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:39.351 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: Deploying daemon mon.c on smithi122 2023-08-25T02:19:39.352 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:39.353 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:39.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/1331185362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:39.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-08-25T02:19:39.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-08-25T02:19:39.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expand map: {default=false} 2023-08-25T02:19:39.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta from 'false' to 'false' 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expanded map: {default=false} 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expand map: {default=info} 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta from 'info' to 'info' 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expanded map: {default=info} 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expand map: {default=daemon} 2023-08-25T02:19:39.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta from 'daemon' to 'daemon' 2023-08-25T02:19:39.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expanded map: {default=daemon} 2023-08-25T02:19:39.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expand map: {default=debug} 2023-08-25T02:19:39.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta from 'debug' to 'debug' 2023-08-25T02:19:39.357 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 ceph-mon[100457]: expand_channel_meta expanded map: {default=debug} 2023-08-25T02:19:39.357 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:38 smithi122 systemd[1]: Started Ceph mon.c for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:19:39.696 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-08-25T02:19:39.696 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph mon dump -f json 2023-08-25T02:19:40.252 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:19:44.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:44.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: mon.a calling monitor election 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: mon.c calling monitor election 2023-08-25T02:19:44.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:44.339 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: fsmap 2023-08-25T02:19:44.340 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:44.340 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: mgrmap e13: x(active, since 40s) 2023-08-25T02:19:44.353 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: overall HEALTH_OK 2023-08-25T02:19:44.355 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.355 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.355 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.356 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:44.356 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:44.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:44.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: mon.a calling monitor election 2023-08-25T02:19:44.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: mon.c calling monitor election 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: fsmap 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: mgrmap e13: x(active, since 40s) 2023-08-25T02:19:44.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: overall HEALTH_OK 2023-08-25T02:19:44.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.493 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:44.494 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:44.494 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:44.496 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 -- 172.21.15.122:0/2003243641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 msgr2=0x7f2428005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:44.496 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 --2- 172.21.15.122:0/2003243641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f2428005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2440068950 tx=0x7f243000feb0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.497 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 -- 172.21.15.122:0/2003243641 shutdown_connections 2023-08-25T02:19:44.497 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 --2- 172.21.15.122:0/2003243641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f2428005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.497 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 -- 172.21.15.122:0/2003243641 >> 172.21.15.122:0/2003243641 conn(0x7f24400faee0 msgr2=0x7f24400fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:44.497 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 -- 172.21.15.122:0/2003243641 shutdown_connections 2023-08-25T02:19:44.498 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.480+0000 7f24463fa700 1 -- 172.21.15.122:0/2003243641 wait complete. 2023-08-25T02:19:44.498 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.481+0000 7f24463fa700 1 Processor -- start 2023-08-25T02:19:44.498 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.481+0000 7f24463fa700 1 -- start start 2023-08-25T02:19:44.498 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f24463fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f24400ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:44.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f24463fa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:44.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f24463fa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440101310 con 0x7f24280035c0 2023-08-25T02:19:44.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f24463fa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24401a1b30 con 0x7f24400ff8d0 2023-08-25T02:19:44.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:44.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:32998/0 (socket says 172.21.15.122:32998) 2023-08-25T02:19:44.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 -- 172.21.15.122:0/556895217 learned_addr learned my addr 172.21.15.122:0/556895217 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:19:44.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 -- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 msgr2=0x7f24400ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-08-25T02:19:44.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 -- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 msgr2=0x7f24400ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-25T02:19:44.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243ffff700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f24400ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:44.508 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T02:19:44.508 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.482+0000 7f243f7fe700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:19:44.512 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243ffff700 1 -- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 msgr2=0x7f24400ffd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:19:44.515 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243ffff700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.516 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243ffff700 1 -- 172.21.15.122:0/556895217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243000f900 con 0x7f24280035c0 2023-08-25T02:19:44.516 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243ffff700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f24400ff390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2430000c00 tx=0x7f24300047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:44.521 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2430013b00 con 0x7f24280035c0 2023-08-25T02:19:44.521 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24300044f0 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24401a1d30 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2430003be0 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.483+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24401a21a0 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.484+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f2430011540 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.484+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2440061f50 con 0x7f24280035c0 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.484+0000 7f243d7fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 0x7f242c03a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:44.522 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.485+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f243004d500 con 0x7f24280035c0 2023-08-25T02:19:44.523 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.485+0000 7f243f7fe700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 0x7f242c03a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:44.523 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.485+0000 7f243f7fe700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 0x7f242c03a4b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2434009180 tx=0x7f2434008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:44.523 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.490+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f24300107e0 con 0x7f24280035c0 2023-08-25T02:19:44.758 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.754+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24401008c0 con 0x7f24280035c0 2023-08-25T02:19:44.758 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.755+0000 7f243d7fa700 1 -- 172.21.15.122:0/556895217 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f243000b2c0 con 0x7f24280035c0 2023-08-25T02:19:44.767 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:19:44.768 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","modified":"2023-08-25T02:19:38.948914Z","created":"2023-08-25T02:18:06.484229Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-08-25T02:19:44.768 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.756+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 msgr2=0x7f242c03a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:44.769 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.756+0000 7f24463fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 0x7f242c03a4b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2434009180 tx=0x7f2434008040 comp rx=0 tx=0).stop 2023-08-25T02:19:44.781 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.756+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 msgr2=0x7f24400ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.756+0000 7f24463fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f24400ff390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2430000c00 tx=0x7f24300047a0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 shutdown_connections 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f242c037ff0 0x7f242c03a4b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24280035c0 0x7f24400ff390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 --2- 172.21.15.122:0/556895217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24400ff8d0 0x7f24400ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 >> 172.21.15.122:0/556895217 conn(0x7f24400faee0 msgr2=0x7f244018e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:44.782 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 shutdown_connections 2023-08-25T02:19:44.783 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:44.757+0000 7f24463fa700 1 -- 172.21.15.122:0/556895217 wait complete. 2023-08-25T02:19:44.783 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2023-08-25T02:19:45.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:45 smithi089 ceph-mon[98732]: Deploying daemon mon.b on smithi121 2023-08-25T02:19:45.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:45 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/556895217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:45.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:45 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:45.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:45 smithi122 ceph-mon[100457]: Deploying daemon mon.b on smithi121 2023-08-25T02:19:45.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:45 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/556895217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:45.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:45 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:46.217 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-08-25T02:19:46.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph mon dump -f json 2023-08-25T02:19:46.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:46 smithi089 ceph-mon[98732]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:46.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:46 smithi122 ceph-mon[100457]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:46.792 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:19:46.907 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:46 smithi121 systemd[1]: Starting Ceph mon.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:19:47.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:46 smithi121 podman[100058]: 2023-08-25T02:19:47.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 bash[100058]: da94d094ec019ab735424aec5a21ca6f87621efad6f15fa54b7ad118532f8a1c 2023-08-25T02:19:47.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: set uid:gid to 167:167 (ceph:ceph) 2023-08-25T02:19:47.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev), process ceph-mon, pid 8 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pidfile_write: ignore empty --pid-file 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: load: jerasure load: lrc 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: RocksDB version: 7.9.2 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Git sha 0 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Compile date 2023-08-24 22:08:31 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: DB SUMMARY 2023-08-25T02:19:47.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: DB Session ID: D527B8VM4KL88JAVJ4ES 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: CURRENT file: CURRENT 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: IDENTITY file: IDENTITY 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.error_if_exists: 0 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.create_if_missing: 0 2023-08-25T02:19:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.paranoid_checks: 1 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.flush_verify_memtable_count: 1 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.env: 0x55cc2450eda0 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.fs: PosixFileSystem 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.info_log: 0x55cc26d5f440 2023-08-25T02:19:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_file_opening_threads: 16 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.statistics: (nil) 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.use_fsync: 0 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_log_file_size: 0 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.log_file_time_to_roll: 0 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.keep_log_file_num: 1000 2023-08-25T02:19:47.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.recycle_log_file_num: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_fallocate: 1 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_mmap_reads: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_mmap_writes: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.use_direct_reads: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.create_missing_column_families: 0 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.db_log_dir: 2023-08-25T02:19:47.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.wal_dir: 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.table_cache_numshardbits: 6 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.WAL_ttl_seconds: 0 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.WAL_size_limit_MB: 0 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.is_fd_close_on_exec: 1 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.advise_random_on_open: 1 2023-08-25T02:19:47.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.db_write_buffer_size: 0 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.write_buffer_manager: 0x55cc26d194a0 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.use_adaptive_mutex: 0 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.rate_limiter: (nil) 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-08-25T02:19:47.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.wal_recovery_mode: 2 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enable_thread_tracking: 0 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enable_pipelined_write: 0 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.unordered_write: 0 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-08-25T02:19:47.177 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.row_cache: None 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.wal_filter: None 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_ingest_behind: 0 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.two_write_queues: 0 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.manual_wal_flush: 0 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.wal_compression: 0 2023-08-25T02:19:47.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.atomic_flush: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.persist_stats_to_disk: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.write_dbid_to_manifest: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.log_readahead_size: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.best_efforts_recovery: 0 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-08-25T02:19:47.179 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.allow_data_in_errors: 0 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.db_host_id: __hostname__ 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enforce_single_del_contracts: true 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_background_jobs: 2 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_background_compactions: -1 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_subcompactions: 1 2023-08-25T02:19:47.180 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.delayed_write_rate : 16777216 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_total_wal_size: 0 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.stats_dump_period_sec: 600 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.stats_persist_period_sec: 600 2023-08-25T02:19:47.181 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_open_files: -1 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bytes_per_sync: 0 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.wal_bytes_per_sync: 0 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.strict_bytes_per_sync: 0 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_readahead_size: 0 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_background_flushes: -1 2023-08-25T02:19:47.182 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Compression algorithms supported: 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kZSTD supported: 0 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kXpressCompression supported: 0 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kLZ4HCCompression supported: 1 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kZlibCompression supported: 1 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kSnappyCompression supported: 1 2023-08-25T02:19:47.183 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kLZ4Compression supported: 1 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: kBZip2Compression supported: 0 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Fast CRC32 supported: Supported on x86 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: DMutex implementation: pthread_mutex_t 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-08-25T02:19:47.184 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.merge_operator: 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_filter: None 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_filter_factory: None 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.sst_partitioner_factory: None 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.memtable_factory: SkipListFactory 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.table_factory: BlockBasedTable 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cc26d5f540) 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout: cache_index_and_filter_blocks: 1 2023-08-25T02:19:47.185 INFO:journalctl@ceph.mon.b.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: pin_top_level_index_and_filter: 1 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_type: 0 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: data_block_index_type: 0 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_shortening: 1 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2023-08-25T02:19:47.186 INFO:journalctl@ceph.mon.b.smithi121.stdout: checksum: 4 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: no_block_cache: 0 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache: 0x55cc26db71f0 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_name: BinnedLRUCache 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_options: 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: capacity : 536870912 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: num_shard_bits : 4 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: strict_capacity_limit : 0 2023-08-25T02:19:47.187 INFO:journalctl@ceph.mon.b.smithi121.stdout: high_pri_pool_ratio: 0.000 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_compressed: (nil) 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: persistent_cache: (nil) 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_size: 4096 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_size_deviation: 10 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_restart_interval: 16 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_block_restart_interval: 1 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: metadata_block_size: 4096 2023-08-25T02:19:47.188 INFO:journalctl@ceph.mon.b.smithi121.stdout: partition_filters: 0 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: use_delta_encoding: 1 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: filter_policy: bloomfilter 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: whole_key_filtering: 1 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: verify_compression: 0 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: read_amp_bytes_per_bit: 0 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: format_version: 5 2023-08-25T02:19:47.189 INFO:journalctl@ceph.mon.b.smithi121.stdout: enable_index_compression: 1 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_align: 0 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout: max_auto_readahead_size: 262144 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout: prepopulate_block_cache: 0 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout: initial_auto_readahead_size: 8192 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.write_buffer_size: 33554432 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_write_buffer_number: 2 2023-08-25T02:19:47.190 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression: NoCompression 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression: Disabled 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.prefix_extractor: nullptr 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.num_levels: 7 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-08-25T02:19:47.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-08-25T02:19:47.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-08-25T02:19:47.192 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-08-25T02:19:47.193 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:19:47.194 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.window_bits: -14 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.level: 32767 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.strategy: 0 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.enabled: false 2023-08-25T02:19:47.195 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.target_file_size_base: 67108864 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.target_file_size_multiplier: 1 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-08-25T02:19:47.196 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-08-25T02:19:47.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.arena_block_size: 1048576 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-08-25T02:19:47.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.disable_auto_compactions: 0 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-08-25T02:19:47.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.table_properties_collectors: 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.inplace_update_support: 0 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.inplace_update_num_locks: 10000 2023-08-25T02:19:47.200 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.memtable_huge_page_size: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.bloom_locality: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.max_successive_merges: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.optimize_filters_for_hits: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.paranoid_file_checks: 0 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.force_consistency_checks: 1 2023-08-25T02:19:47.201 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.report_bg_io_stats: 0 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.ttl: 2592000 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.periodic_compaction_seconds: 0 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enable_blob_files: false 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.min_blob_size: 0 2023-08-25T02:19:47.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_file_size: 268435456 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_compression_type: NoCompression 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.enable_blob_garbage_collection: false 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.blob_file_starting_level: 0 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-08-25T02:19:47.203 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-08-25T02:19:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-08-25T02:19:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 77695987-5614-401c-9251-2826fff16208 2023-08-25T02:19:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929987121236, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-08-25T02:19:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-08-25T02:19:47.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929987121639, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1692929987, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "77695987-5614-401c-9251-2826fff16208", "db_session_id": "D527B8VM4KL88JAVJ4ES", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1692929987121705, "job": 1, "event": "recovery_finished"} 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cc26e54000 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: DB pointer 0x55cc26e40000 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-08-25T02:19:47.205 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** DB Stats ** 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-08-25T02:19:47.206 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** Compaction Stats [default] ** 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:47.207 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** Compaction Stats [default] ** 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-08-25T02:19:47.208 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:19:47.209 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout: Block cache BinnedLRUCache@0x55cc26db71f0#8 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-08-25T02:19:47.210 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: using public_addr v2:172.21.15.121:0/0 -> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: starting mon.b rank -1 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(???) e0 preinit fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).mds e1 new map 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).mds e1 print_map 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout: e1 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-08-25T02:19:47.211 INFO:journalctl@ceph.mon.b.smithi121.stdout: legacy client fscid: -1 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout: No filesystems configured 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-08-25T02:19:47.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-08-25T02:19:47.213 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:19:47.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/442069125' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:47.216 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c;count:3 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:47.217 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Deploying daemon mon.c on smithi122 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/1331185362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.a calling monitor election 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.c calling monitor election 2023-08-25T02:19:47.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:47.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: monmap e2: 2 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: fsmap 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mgrmap e13: x(active, since 40s) 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: overall HEALTH_OK 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: Deploying daemon mon.b on smithi121 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/556895217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-08-25T02:19:47.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expand map: {default=false} 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta from 'false' to 'false' 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expanded map: {default=false} 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expand map: {default=info} 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta from 'info' to 'info' 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expanded map: {default=info} 2023-08-25T02:19:47.224 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expand map: {default=daemon} 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta from 'daemon' to 'daemon' 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expanded map: {default=daemon} 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expand map: {default=debug} 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta from 'debug' to 'debug' 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 ceph-mon[100099]: expand_channel_meta expanded map: {default=debug} 2023-08-25T02:19:47.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:47 smithi121 systemd[1]: Started Ceph mon.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:19:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.499+0000 7ffa43fff700 1 -- 172.21.15.122:0/437061801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0035c0 msgr2=0x7ffa2c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 --2- 172.21.15.122:0/437061801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0035c0 0x7ffa2c005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa440ff120 tx=0x7ffa3800fe60 comp rx=0 tx=0).stop 2023-08-25T02:19:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 -- 172.21.15.122:0/437061801 shutdown_connections 2023-08-25T02:19:52.501 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 --2- 172.21.15.122:0/437061801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0035c0 0x7ffa2c005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 -- 172.21.15.122:0/437061801 >> 172.21.15.122:0/437061801 conn(0x7ffa440fad00 msgr2=0x7ffa440fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 -- 172.21.15.122:0/437061801 shutdown_connections 2023-08-25T02:19:52.502 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.500+0000 7ffa43fff700 1 -- 172.21.15.122:0/437061801 wait complete. 2023-08-25T02:19:52.504 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 Processor -- start 2023-08-25T02:19:52.504 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 -- start start 2023-08-25T02:19:52.504 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:52.504 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:52.504 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 0x7ffa44196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4419c2a0 con 0x7ffa44195cb0 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa441a2510 con 0x7ffa4419bc30 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa43fff700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa441a2680 con 0x7ffa2c0035c0 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33024/0 (socket says 172.21.15.122:33024) 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 -- 172.21.15.122:0/497406144 learned_addr learned my addr 172.21.15.122:0/497406144 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:19:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa427fc700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 msgr2=0x7ffa4419b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 msgr2=0x7ffa4419b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.501+0000 7ffa42ffd700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 0x7ffa44196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa427fc700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 msgr2=0x7ffa44195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa427fc700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 msgr2=0x7ffa44195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-25T02:19:52.506 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa42ffd700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:19:52.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa427fc700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T02:19:52.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa427fc700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:19:52.507 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 msgr2=0x7ffa4419b6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:19:52.508 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.508 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 msgr2=0x7ffa44195770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:19:52.508 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 -- 172.21.15.122:0/497406144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3800f8b0 con 0x7ffa44195cb0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa437fe700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 0x7ffa44196130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00bd50 tx=0x7ffa3c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c00cd10 con 0x7ffa44195cb0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa3c00ce70 con 0x7ffa44195cb0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa441a2960 con 0x7ffa44195cb0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c012840 con 0x7ffa44195cb0 2023-08-25T02:19:52.509 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.502+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa441a2e60 con 0x7ffa44195cb0 2023-08-25T02:19:52.510 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.503+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7ffa3c0129a0 con 0x7ffa44195cb0 2023-08-25T02:19:52.510 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.503+0000 7ffa2bfff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 0x7ffa3003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:52.516 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.503+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffa3c04a5d0 con 0x7ffa44195cb0 2023-08-25T02:19:52.517 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.504+0000 7ffa42ffd700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 0x7ffa3003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:52.517 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.504+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa44061f50 con 0x7ffa44195cb0 2023-08-25T02:19:52.517 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.504+0000 7ffa42ffd700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 0x7ffa3003aaf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa38020040 tx=0x7ffa38001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:52.517 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.509+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ffa3c01d3b0 con 0x7ffa44195cb0 2023-08-25T02:19:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: mon.c calling monitor election 2023-08-25T02:19:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: mon.a calling monitor election 2023-08-25T02:19:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:52.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: fsmap 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: mgrmap e13: x(active, since 48s) 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-08-25T02:19:52.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2023-08-25T02:19:52.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: mon.c calling monitor election 2023-08-25T02:19:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: mon.a calling monitor election 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: fsmap 2023-08-25T02:19:52.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: mgrmap e13: x(active, since 48s) 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:52.793 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.788+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa44106f70 con 0x7ffa44195cb0 2023-08-25T02:19:52.793 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.788+0000 7ffa2bfff700 1 -- 172.21.15.122:0/497406144 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7ffa3c018170 con 0x7ffa44195cb0 2023-08-25T02:19:52.793 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-08-25T02:19:52.793 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","modified":"2023-08-25T02:19:47.148381Z","created":"2023-08-25T02:18:06.484229Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-08-25T02:19:52.794 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.790+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 msgr2=0x7ffa3003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:52.794 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.790+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 0x7ffa3003aaf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffa38020040 tx=0x7ffa38001e70 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 msgr2=0x7ffa44196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 0x7ffa44196130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00bd50 tx=0x7ffa3c00be30 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 shutdown_connections 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa44195cb0 0x7ffa44196130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa2c0035c0 0x7ffa4419b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ffa30038630 0x7ffa3003aaf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 --2- 172.21.15.122:0/497406144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4419bc30 0x7ffa44195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:52.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 >> 172.21.15.122:0/497406144 conn(0x7ffa440fad00 msgr2=0x7ffa44102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:52.796 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 shutdown_connections 2023-08-25T02:19:52.796 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:52.791+0000 7ffa43fff700 1 -- 172.21.15.122:0/497406144 wait complete. 2023-08-25T02:19:52.798 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 3 2023-08-25T02:19:53.248 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-08-25T02:19:53.248 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph config generate-minimal-conf 2023-08-25T02:19:53.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:53 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/497406144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:53 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/497406144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:53.817 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:19:54.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: mon.c calling monitor election 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: mon.a calling monitor election 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:54.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-08-25T02:19:54.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: fsmap 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: mgrmap e13: x(active, since 48s) 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2023-08-25T02:19:54.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/497406144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-08-25T02:19:54.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:54 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:54.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 -- 172.21.15.89:0/2029572109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c106440 msgr2=0x7fec7c106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:54.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 --2- 172.21.15.89:0/2029572109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c106440 0x7fec7c106820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fec6c009910 tx=0x7fec6c009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:54.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 -- 172.21.15.89:0/2029572109 shutdown_connections 2023-08-25T02:19:54.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 --2- 172.21.15.89:0/2029572109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c106440 0x7fec7c106820 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:54.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 -- 172.21.15.89:0/2029572109 >> 172.21.15.89:0/2029572109 conn(0x7fec7c074e20 msgr2=0x7fec7c075230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:54.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 -- 172.21.15.89:0/2029572109 shutdown_connections 2023-08-25T02:19:54.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.919+0000 7fec80c7f700 1 -- 172.21.15.89:0/2029572109 wait complete. 2023-08-25T02:19:54.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 Processor -- start 2023-08-25T02:19:54.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 -- start start 2023-08-25T02:19:54.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7c106440 0x7fec7c0ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:54.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 0x7fec7c1a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1a45d0 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1a4740 con 0x7fec7c1a1b90 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec80c7f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1a4900 con 0x7fec7c106440 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec79d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec79d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45770/0 (socket says 172.21.15.89:45770) 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec79d9b700 1 -- 172.21.15.89:0/3823122998 learned_addr learned my addr 172.21.15.89:0/3823122998 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec7a59c700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7c106440 0x7fec7c0ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.920+0000 7fec7ad9d700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 0x7fec7c1a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 -- 172.21.15.89:0/3823122998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7c106440 msgr2=0x7fec7c0ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7c106440 0x7fec7c0ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 -- 172.21.15.89:0/3823122998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 msgr2=0x7fec7c1a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 0x7fec7c1a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:54.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 -- 172.21.15.89:0/3823122998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0095f0 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec7ad9d700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 0x7fec7c1a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:19:54.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec79d9b700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b780 tx=0x7fec7000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:54.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70004230 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c1a4be0 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec70004390 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70011ad0 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.921+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c1a50e0 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7fec70011c30 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec6b7fe700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 0x7fec6403aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:54.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec7004b470 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec7a59c700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 0x7fec6403aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:54.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c061f50 con 0x7fec7c0ff8d0 2023-08-25T02:19:54.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.922+0000 7fec7a59c700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 0x7fec6403aaf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec6c006010 tx=0x7fec6c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:54.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:54.928+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fec700193a0 con 0x7fec7c0ff8d0 2023-08-25T02:19:55.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.123+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fec7c10a740 con 0x7fec7c0ff8d0 2023-08-25T02:19:55.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.124+0000 7fec6b7fe700 1 -- 172.21.15.89:0/3823122998 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fec70048030 con 0x7fec7c0ff8d0 2023-08-25T02:19:55.126 INFO:teuthology.orchestra.run.smithi089.stdout:# minimal ceph.conf for 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:55.126 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2023-08-25T02:19:55.126 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:19:55.126 INFO:teuthology.orchestra.run.smithi089.stdout: mon_host = [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2023-08-25T02:19:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 msgr2=0x7fec6403aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 0x7fec6403aaf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec6c006010 tx=0x7fec6c00b540 comp rx=0 tx=0).stop 2023-08-25T02:19:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 msgr2=0x7fec7c0ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b780 tx=0x7fec7000ba90 comp rx=0 tx=0).stop 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 shutdown_connections 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec64038630 0x7fec6403aaf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7c1a1b90 0x7fec7c1a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7c106440 0x7fec7c0ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 --2- 172.21.15.89:0/3823122998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec7c0ff8d0 0x7fec7c0ffd50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 >> 172.21.15.89:0/3823122998 conn(0x7fec7c074e20 msgr2=0x7fec7c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 shutdown_connections 2023-08-25T02:19:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:19:55.126+0000 7fec80c7f700 1 -- 172.21.15.89:0/3823122998 wait complete. 2023-08-25T02:19:55.315 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mon.b calling monitor election 2023-08-25T02:19:55.315 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:19:55.315 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:19:55.315 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:55.316 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mon.b calling monitor election 2023-08-25T02:19:55.316 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mon.c calling monitor election 2023-08-25T02:19:55.316 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mon.a calling monitor election 2023-08-25T02:19:55.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-08-25T02:19:55.317 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:55.320 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: fsmap 2023-08-25T02:19:55.320 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:55.320 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: mgrmap e13: x(active, since 50s) 2023-08-25T02:19:55.321 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-08-25T02:19:55.321 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:19:55.321 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: overall HEALTH_OK 2023-08-25T02:19:55.321 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.323 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.323 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.323 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:55.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:55.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:55.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mon.b calling monitor election 2023-08-25T02:19:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:19:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:19:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mon.b calling monitor election 2023-08-25T02:19:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mon.c calling monitor election 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mon.a calling monitor election 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: fsmap 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: mgrmap e13: x(active, since 50s) 2023-08-25T02:19:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: overall HEALTH_OK 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.510 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.510 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.510 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:55.511 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:55.511 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:55.511 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mon.b calling monitor election 2023-08-25T02:19:55.512 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:19:55.513 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:19:55.513 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:19:55.513 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mon.b calling monitor election 2023-08-25T02:19:55.513 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mon.c calling monitor election 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mon.a calling monitor election 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: monmap e3: 3 mons at {a=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: fsmap 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: osdmap e4: 0 total, 0 up, 0 in 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: mgrmap e13: x(active, since 50s) 2023-08-25T02:19:55.514 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: overall HEALTH_OK 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.515 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.516 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.516 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.531 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.532 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.532 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:55.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:55.863 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-08-25T02:19:55.863 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:55.864 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2023-08-25T02:19:55.897 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:55.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:55.968 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:19:55.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2023-08-25T02:19:56.009 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:19:56.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:56.080 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:19:56.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2023-08-25T02:19:56.126 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:19:56.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:19:56.150 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: Reconfiguring mon.a (unknown last config time)... 2023-08-25T02:19:56.150 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: Reconfiguring daemon mon.a on smithi089 2023-08-25T02:19:56.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3823122998' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.226 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:56.227 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:56.228 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.228 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.228 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:56.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:56.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.264 INFO:tasks.cephadm:Adding mgr.x on smithi089 2023-08-25T02:19:56.265 INFO:tasks.cephadm:Adding mgr.y on smithi121 2023-08-25T02:19:56.265 INFO:tasks.cephadm:Adding mgr.z on smithi122 2023-08-25T02:19:56.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch apply mgr '3;smithi089=x;smithi121=y;smithi122=z' 2023-08-25T02:19:56.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: Reconfiguring mon.a (unknown last config time)... 2023-08-25T02:19:56.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: Reconfiguring daemon mon.a on smithi089 2023-08-25T02:19:56.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3823122998' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:56.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: Reconfiguring mon.a (unknown last config time)... 2023-08-25T02:19:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: Reconfiguring daemon mon.a on smithi089 2023-08-25T02:19:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3823122998' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:19:56.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:56.580 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:19:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:19:56.147+0000 7fa1a1b37700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-08-25T02:19:56.885 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:19:57.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:57 smithi121 ceph-mon[100099]: Reconfiguring mon.b (monmap changed)... 2023-08-25T02:19:57.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:57 smithi121 ceph-mon[100099]: Reconfiguring daemon mon.b on smithi121 2023-08-25T02:19:57.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:57 smithi089 ceph-mon[98732]: Reconfiguring mon.b (monmap changed)... 2023-08-25T02:19:57.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:57 smithi089 ceph-mon[98732]: Reconfiguring daemon mon.b on smithi121 2023-08-25T02:19:57.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:57 smithi122 ceph-mon[100457]: Reconfiguring mon.b (monmap changed)... 2023-08-25T02:19:57.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:57 smithi122 ceph-mon[100457]: Reconfiguring daemon mon.b on smithi121 2023-08-25T02:19:58.336 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 -- 172.21.15.122:0/267268136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31000ff430 msgr2=0x7f31000ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:58.336 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 --2- 172.21.15.122:0/267268136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31000ff430 0x7f31000ff850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009910 tx=0x7f30f8009c20 comp rx=0 tx=0).stop 2023-08-25T02:19:58.336 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 -- 172.21.15.122:0/267268136 shutdown_connections 2023-08-25T02:19:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 --2- 172.21.15.122:0/267268136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31000ff430 0x7f31000ff850 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 -- 172.21.15.122:0/267268136 >> 172.21.15.122:0/267268136 conn(0x7f31000fa9f0 msgr2=0x7f31000fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 -- 172.21.15.122:0/267268136 shutdown_connections 2023-08-25T02:19:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 -- 172.21.15.122:0/267268136 wait complete. 2023-08-25T02:19:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.335+0000 7f31050dc700 1 Processor -- start 2023-08-25T02:19:58.339 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 -- start start 2023-08-25T02:19:58.339 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 0x7f3100199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3100195040 0x7f310019c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310019c960 con 0x7f3100195040 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310019cad0 con 0x7f3100194bc0 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f31050dc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310019cc90 con 0x7f31000ff430 2023-08-25T02:19:58.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33034/0 (socket says 172.21.15.122:33034) 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 -- 172.21.15.122:0/2607685228 learned_addr learned my addr 172.21.15.122:0/2607685228 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30ff59e700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3100195040 0x7f310019c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fe59c700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 0x7f3100199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 -- 172.21.15.122:0/2607685228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 msgr2=0x7f3100199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 0x7f3100199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 -- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3100195040 msgr2=0x7f310019c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:58.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3100195040 0x7f310019c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 -- 172.21.15.122:0/2607685228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f80095f0 con 0x7f31000ff430 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fe59c700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 0x7f3100199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.336+0000 7f30fed9d700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000c00 tx=0x7f30f8004cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f801d070 con 0x7f31000ff430 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30f800be20 con 0x7f31000ff430 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3100068140 con 0x7f31000ff430 2023-08-25T02:19:58.342 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800f460 con 0x7f31000ff430 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3100068660 con 0x7f31000ff430 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.337+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f30f800f5c0 con 0x7f31000ff430 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.338+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3100105130 con 0x7f31000ff430 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.338+0000 7f30e7fff700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 0x7f30e803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.338+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f30f804c160 con 0x7f31000ff430 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.338+0000 7f30fe59c700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 0x7f30e803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:19:58.343 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.338+0000 7f30fe59c700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 0x7f30e803aaf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30f40079b0 tx=0x7f30f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:19:58.344 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.343+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f30f8020960 con 0x7f31000ff430 2023-08-25T02:19:58.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: Reconfiguring mon.c (monmap changed)... 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:58.375 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:58.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:58 smithi122 ceph-mon[100457]: Reconfiguring daemon mon.c on smithi122 2023-08-25T02:19:58.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: Reconfiguring mon.c (monmap changed)... 2023-08-25T02:19:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:58.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:58 smithi121 ceph-mon[100099]: Reconfiguring daemon mon.c on smithi122 2023-08-25T02:19:58.552 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.552+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi121=y;smithi122=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f310018dda0 con 0x7f30e8038630 2023-08-25T02:19:58.562 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.558+0000 7f30e7fff700 1 -- 172.21.15.122:0/2607685228 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f310018dda0 con 0x7f30e8038630 2023-08-25T02:19:58.563 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mgr update... 2023-08-25T02:19:58.564 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 msgr2=0x7f30e803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 0x7f30e803aaf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30f40079b0 tx=0x7f30f4008040 comp rx=0 tx=0).stop 2023-08-25T02:19:58.565 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 msgr2=0x7f3100194680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30f8000c00 tx=0x7f30f8004cb0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 shutdown_connections 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3100195040 0x7f310019c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31000ff430 0x7f3100194680 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f30e8038630 0x7f30e803aaf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 --2- 172.21.15.122:0/2607685228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3100194bc0 0x7f3100199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:19:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 >> 172.21.15.122:0/2607685228 conn(0x7f31000fa9f0 msgr2=0x7f31000fb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:19:58.567 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 shutdown_connections 2023-08-25T02:19:58.567 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:19:58.560+0000 7f31050dc700 1 -- 172.21.15.122:0/2607685228 wait complete. 2023-08-25T02:19:58.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: Reconfiguring mon.c (monmap changed)... 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:58 smithi089 ceph-mon[98732]: Reconfiguring daemon mon.c on smithi122 2023-08-25T02:19:59.238 DEBUG:teuthology.orchestra.run.smithi121:mgr.y> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.y.service 2023-08-25T02:19:59.243 DEBUG:teuthology.orchestra.run.smithi122:mgr.z> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.z.service 2023-08-25T02:19:59.276 INFO:journalctl@ceph.mgr.y.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:19:59.284 INFO:tasks.cephadm:Deploying OSDs... 2023-08-25T02:19:59.285 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:19:59.285 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2023-08-25T02:19:59.307 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-25T02:19:59.307 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2023-08-25T02:19:59.317 INFO:journalctl@ceph.mgr.z.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:19:59.381 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-25T02:19:59.381 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 386428 Links: 1 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-08-25 02:19:12.522978525 +0000 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-08-25 02:12:56.878424809 +0000 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-08-25 02:12:56.878424809 +0000 2023-08-25T02:19:59.382 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-08-25T02:19:59.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-25T02:19:59.455 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-08-25T02:19:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-08-25T02:19:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000255649 s, 2.0 MB/s 2023-08-25T02:19:59.459 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-25T02:19:59.549 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2023-08-25T02:19:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-25T02:19:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:19:59.622 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 392726 Links: 1 2023-08-25T02:19:59.622 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:19:59.623 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:19:59.645 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-08-25 02:19:12.522978525 +0000 2023-08-25T02:19:59.646 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-08-25 02:12:57.424412521 +0000 2023-08-25T02:19:59.646 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-08-25 02:12:57.424412521 +0000 2023-08-25T02:19:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-08-25T02:19:59.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-25T02:19:59.690 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-08-25T02:19:59.691 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-08-25T02:19:59.691 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000236283 s, 2.2 MB/s 2023-08-25T02:19:59.691 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-25T02:19:59.751 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2023-08-25T02:19:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-25T02:19:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:19:59.811 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 389665 Links: 1 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-08-25 02:19:12.522978525 +0000 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-08-25 02:12:57.938400953 +0000 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-08-25 02:12:57.938400953 +0000 2023-08-25T02:19:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-08-25T02:19:59.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-25T02:19:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi121=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: Saving service mgr spec with placement smithi089=x;smithi121=y;smithi122=z;count:3 2023-08-25T02:19:59.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:59.838 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.839 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:19:59.839 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:19:59.839 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:19:59.839 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:19:59 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-08-25T02:19:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-08-25T02:19:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000253877 s, 2.0 MB/s 2023-08-25T02:19:59.876 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-25T02:19:59.935 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2023-08-25T02:19:59.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi121=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:59.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: Saving service mgr spec with placement smithi089=x;smithi121=y;smithi122=z;count:3 2023-08-25T02:19:59.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:19:59.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:59.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:19:59.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.978 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:19:59.978 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:19:59.978 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:19:59.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:19:59 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:19:59.989 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi121=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:19:59.990 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: Saving service mgr spec with placement smithi089=x;smithi121=y;smithi122=z;count:3 2023-08-25T02:19:59.990 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.991 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:19:59.993 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:00.009 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:00.009 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:00.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:19:59 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:00.012 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 392044 Links: 1 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-08-25 02:19:12.522978525 +0000 2023-08-25T02:20:00.013 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-08-25 02:12:58.448389475 +0000 2023-08-25T02:20:00.014 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-08-25 02:12:58.448389475 +0000 2023-08-25T02:20:00.014 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-08-25T02:20:00.014 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-25T02:20:00.085 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-08-25T02:20:00.086 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-08-25T02:20:00.086 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000209227 s, 2.4 MB/s 2023-08-25T02:20:00.086 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-25T02:20:00.147 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-08-25T02:20:00.148 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-08-25T02:20:00.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-25T02:20:00.172 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-08-25T02:20:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-25T02:20:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 389460 Links: 1 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-08-25 02:19:46.538535177 +0000 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-08-25 02:13:01.482581495 +0000 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-08-25 02:13:01.482581495 +0000 2023-08-25T02:20:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-08-25T02:20:00.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-25T02:20:00.309 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-08-25T02:20:00.309 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-08-25T02:20:00.309 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000226317 s, 2.3 MB/s 2023-08-25T02:20:00.310 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-25T02:20:00.373 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-08-25T02:20:00.437 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 389622 Links: 1 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-08-25 02:19:46.538535177 +0000 2023-08-25T02:20:00.438 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-08-25 02:13:02.032574699 +0000 2023-08-25T02:20:00.439 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-08-25 02:13:02.032574699 +0000 2023-08-25T02:20:00.459 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-08-25T02:20:00.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-25T02:20:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-08-25T02:20:00.508 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-08-25T02:20:00.508 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000231332 s, 2.2 MB/s 2023-08-25T02:20:00.509 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-25T02:20:00.573 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-08-25T02:20:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-25T02:20:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:00.646 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 389675 Links: 1 2023-08-25T02:20:00.646 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:00.646 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:00.647 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-08-25 02:19:46.538535177 +0000 2023-08-25T02:20:00.648 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-08-25 02:13:02.588567827 +0000 2023-08-25T02:20:00.648 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-08-25 02:13:02.588567827 +0000 2023-08-25T02:20:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-08-25T02:20:00.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-25T02:20:00.716 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-08-25T02:20:00.717 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-08-25T02:20:00.717 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000187712 s, 2.7 MB/s 2023-08-25T02:20:00.717 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-25T02:20:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:00 smithi122 ceph-mon[100457]: Deploying daemon mgr.z on smithi122 2023-08-25T02:20:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:00 smithi122 ceph-mon[100457]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:00 smithi122 ceph-mon[100457]: overall HEALTH_OK 2023-08-25T02:20:00.729 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:00 smithi122 systemd[1]: Starting Ceph mgr.z for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:20:00.778 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-08-25T02:20:00.836 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:00 smithi089 ceph-mon[98732]: Deploying daemon mgr.z on smithi122 2023-08-25T02:20:00.837 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:00 smithi089 ceph-mon[98732]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:00.837 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:00 smithi089 ceph-mon[98732]: overall HEALTH_OK 2023-08-25T02:20:00.847 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-25T02:20:00.848 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:00.848 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 389768 Links: 1 2023-08-25T02:20:00.848 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:00.848 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:00.848 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-08-25 02:19:46.538535177 +0000 2023-08-25T02:20:00.849 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-08-25 02:13:03.117561290 +0000 2023-08-25T02:20:00.849 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-08-25 02:13:03.117561290 +0000 2023-08-25T02:20:00.849 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-08-25T02:20:00.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-25T02:20:00.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:00 smithi121 ceph-mon[100099]: Deploying daemon mgr.z on smithi122 2023-08-25T02:20:00.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:00 smithi121 ceph-mon[100099]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:00.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:00 smithi121 ceph-mon[100099]: overall HEALTH_OK 2023-08-25T02:20:00.915 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-08-25T02:20:00.917 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-08-25T02:20:00.918 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000223595 s, 2.3 MB/s 2023-08-25T02:20:00.918 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-25T02:20:00.977 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-08-25T02:20:00.977 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2023-08-25T02:20:00.999 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-25T02:20:01.000 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2023-08-25T02:20:01.058 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-25T02:20:01.059 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:01.059 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 382728 Links: 1 2023-08-25T02:20:01.059 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:01.060 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:01.063 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-08-25 02:20:00.603333150 +0000 2023-08-25T02:20:01.063 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-08-25 02:13:11.259416084 +0000 2023-08-25T02:20:01.063 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-08-25 02:13:11.259416084 +0000 2023-08-25T02:20:01.078 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-08-25T02:20:01.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-25T02:20:01.128 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-08-25T02:20:01.128 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-08-25T02:20:01.128 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000231795 s, 2.2 MB/s 2023-08-25T02:20:01.130 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-25T02:20:01.190 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2023-08-25T02:20:01.225 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:00 smithi122 podman[102283]: 2023-08-25T02:20:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-25T02:20:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:01.247 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 387227 Links: 1 2023-08-25T02:20:01.247 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:01.247 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:01.248 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-08-25 02:20:00.603333150 +0000 2023-08-25T02:20:01.248 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-08-25 02:13:11.800403766 +0000 2023-08-25T02:20:01.252 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-08-25 02:13:11.800403766 +0000 2023-08-25T02:20:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-08-25T02:20:01.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-25T02:20:01.317 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-08-25T02:20:01.318 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-08-25T02:20:01.318 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000210113 s, 2.4 MB/s 2023-08-25T02:20:01.318 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-25T02:20:01.393 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2023-08-25T02:20:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 391465 Links: 1 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-08-25 02:20:00.603333150 +0000 2023-08-25T02:20:01.466 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-08-25 02:13:12.391390309 +0000 2023-08-25T02:20:01.467 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-08-25 02:13:12.391390309 +0000 2023-08-25T02:20:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-08-25T02:20:01.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-25T02:20:01.511 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 bash[102283]: 1dd3fb09857efaa4764c7bbe2848542e9dca3d2130c04c73ba9217e316c8ebaf 2023-08-25T02:20:01.512 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-mgr[102405]: -- 172.21.15.122:0/2960328287 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56045b0231e0 con 0x56045aff8c00 2023-08-25T02:20:01.512 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 systemd[1]: Started Ceph mgr.z for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:20:01.521 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:01.350+0000 7f3e2f446200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:20:01.521 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:01.433+0000 7f3e2f446200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:20:01.550 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-08-25T02:20:01.550 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-08-25T02:20:01.553 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000203323 s, 2.5 MB/s 2023-08-25T02:20:01.554 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-25T02:20:01.613 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2023-08-25T02:20:01.680 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-25T02:20:01.683 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T02:20:01.686 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 392462 Links: 1 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-08-25 02:20:00.603333150 +0000 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-08-25 02:13:12.939377831 +0000 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-08-25 02:13:12.939377831 +0000 2023-08-25T02:20:01.687 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-08-25T02:20:01.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-25T02:20:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-08-25T02:20:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-08-25T02:20:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000232174 s, 2.2 MB/s 2023-08-25T02:20:01.752 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-25T02:20:01.805 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:01.525+0000 7f3e2f446200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:20:01.807 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:01.625+0000 7f3e2f446200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:20:01.815 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:01.801+0000 7f3e2f446200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:20:01.821 INFO:tasks.cephadm:Deploying osd.0 on smithi089 with /dev/vg_nvme/lv_4... 2023-08-25T02:20:01.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-08-25T02:20:02.391 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:20:02.478 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:02.165+0000 7f3e2f446200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:20:02.479 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:02.299+0000 7f3e2f446200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:20:02.479 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:02.386+0000 7f3e2f446200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:20:02.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:02.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:02.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:20:02.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:02.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:02.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-mon[100457]: Deploying daemon mgr.y on smithi121 2023-08-25T02:20:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:02.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:02.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:02 smithi121 ceph-mon[100099]: Deploying daemon mgr.y on smithi121 2023-08-25T02:20:02.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:02.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:02.564 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:02.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-08-25T02:20:02.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:02.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:02.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:02 smithi089 ceph-mon[98732]: Deploying daemon mgr.y on smithi121 2023-08-25T02:20:02.763 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:02 smithi121 systemd[1]: Starting Ceph mgr.y for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:20:02.898 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:02.479+0000 7f3e2f446200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:20:03.228 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:02.897+0000 7f3e2f446200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:20:03.347 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 podman[101543]: 2023-08-25T02:20:03.347 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 bash[101543]: e66daba24690ff2bdeb963b61fc214036222e3943b02739e4b82795ab08e7742 2023-08-25T02:20:03.348 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-mgr[101583]: -- 172.21.15.121:0/979805412 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555f20bc31e0 con 0x555f20b98c00 2023-08-25T02:20:03.607 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 systemd[1]: Started Ceph mgr.y for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:20:03.608 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:03.426+0000 7fc8936e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:20:03.608 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:03.511+0000 7fc8936e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:20:03.608 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:03.606+0000 7fc8936e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-08-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-08-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-08-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-08-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111186 s, 943 MB/s 2023-08-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-08-25T02:20:03.887 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:03.708+0000 7fc8936e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:20:03.978 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:03.561+0000 7f3e2f446200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:20:03.979 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:03.649+0000 7f3e2f446200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:20:04.148 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2023-08-25T02:20:04.220 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:03.884+0000 7fc8936e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:20:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.612 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.612 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:04.612 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:04.285+0000 7fc8936e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:20:04.612 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:04.418+0000 7fc8936e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:20:04.612 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:04.512+0000 7fc8936e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:20:04.703 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:20:04.730 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:04.471+0000 7f3e2f446200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:20:04.731 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:04.567+0000 7f3e2f446200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:20:04.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:04.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:04.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:04.970 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:04.609+0000 7fc8936e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:20:05.052 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:04.758+0000 7f3e2f446200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:20:05.052 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:05.049+0000 7f3e2f446200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:20:05.427 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:05.277+0000 7f3e2f446200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:20:05.469 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:05.070+0000 7fc8936e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:20:05.689 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:05.426+0000 7f3e2f446200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:20:05.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 -- 172.21.15.89:0/2686043295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 msgr2=0x7f6624108e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 --2- 172.21.15.89:0/2686043295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f6624108e20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f661c00b0d0 tx=0x7f661c00b3e0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 -- 172.21.15.89:0/2686043295 shutdown_connections 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 --2- 172.21.15.89:0/2686043295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 0x7f6624100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 --2- 172.21.15.89:0/2686043295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f6624108e20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 --2- 172.21.15.89:0/2686043295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 0x7f6624102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 -- 172.21.15.89:0/2686043295 >> 172.21.15.89:0/2686043295 conn(0x7f66240fbf30 msgr2=0x7f66240fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:20:05.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 -- 172.21.15.89:0/2686043295 shutdown_connections 2023-08-25T02:20:05.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.785+0000 7f662a589700 1 -- 172.21.15.89:0/2686043295 wait complete. 2023-08-25T02:20:05.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 Processor -- start 2023-08-25T02:20:05.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 -- start start 2023-08-25T02:20:05.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 0x7f6624196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:05.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 0x7f6624196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:05.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:05.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419e2f0 con 0x7f6624102570 2023-08-25T02:20:05.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419e430 con 0x7f66241009d0 2023-08-25T02:20:05.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f662a589700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419e5c0 con 0x7f6624101bd0 2023-08-25T02:20:05.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f6628b26700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:05.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f6623fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 0x7f6624196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:05.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f6628b26700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45786/0 (socket says 172.21.15.89:45786) 2023-08-25T02:20:05.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f6628b26700 1 -- 172.21.15.89:0/2274284897 learned_addr learned my addr 172.21.15.89:0/2274284897 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:20:05.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.786+0000 7f66237fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 0x7f6624196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 -- 172.21.15.89:0/2274284897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 msgr2=0x7f6624196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 0x7f6624196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 -- 172.21.15.89:0/2274284897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 msgr2=0x7f6624196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 0x7f6624196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 -- 172.21.15.89:0/2274284897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c009d00 con 0x7f6624102570 2023-08-25T02:20:05.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f6628b26700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f661c0094c0 tx=0x7f661c0094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:05.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00e040 con 0x7f6624102570 2023-08-25T02:20:05.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6624072b50 con 0x7f6624102570 2023-08-25T02:20:05.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f661c003e10 con 0x7f6624102570 2023-08-25T02:20:05.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c01b9e0 con 0x7f6624102570 2023-08-25T02:20:05.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.787+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6624073040 con 0x7f6624102570 2023-08-25T02:20:05.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.788+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45191+0+0 (secure 0 0 0) 0x7f661c01bb40 con 0x7f6624102570 2023-08-25T02:20:05.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.788+0000 7f66217fa700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 0x7f660c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:05.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.788+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f661c04c190 con 0x7f6624102570 2023-08-25T02:20:05.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.788+0000 7f6623fff700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 0x7f660c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:05.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.788+0000 7f6623fff700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 0x7f660c03a8a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6624101a30 tx=0x7f6614007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:05.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.789+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6624107f80 con 0x7f6624102570 2023-08-25T02:20:05.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:05.793+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f661c01f070 con 0x7f6624102570 2023-08-25T02:20:05.979 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:05.687+0000 7f3e2f446200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:20:06.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:06.005+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f662402cbe0 con 0x7f660c0383e0 2023-08-25T02:20:06.109 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:05.735+0000 7fc8936e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:20:06.109 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:05.817+0000 7fc8936e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:20:06.180 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: Reconfiguring mgr.x (unknown last config time)... 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.182 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: Reconfiguring daemon mgr.x on smithi089 2023-08-25T02:20:06.183 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:06.183 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:06.183 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: Reconfiguring mgr.x (unknown last config time)... 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:06.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:06.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: Reconfiguring daemon mgr.x on smithi089 2023-08-25T02:20:06.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:06.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:06.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: Reconfiguring mgr.x (unknown last config time)... 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: Reconfiguring daemon mgr.x on smithi089 2023-08-25T02:20:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:06.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:06.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:06.728 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:06.406+0000 7f3e2f446200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:20:06.728 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:06.539+0000 7f3e2f446200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:20:06.846 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:06.584+0000 7fc8936e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:20:06.846 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:06.674+0000 7fc8936e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:20:07.061 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:06.730+0000 7f3e2f446200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:20:07.068 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:06.842+0000 7f3e2f446200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:20:07.125 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:06.845+0000 7fc8936e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:20:07.389 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:07.124+0000 7fc8936e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:20:07.389 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:07.347+0000 7fc8936e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:20:07.394 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:07.059+0000 7f3e2f446200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:20:07.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:07.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.587 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:07.590 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:07.590 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:07.594 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.721 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:07.495+0000 7fc8936e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:20:07.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:07.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:07.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:07.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:07.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.735 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:07.736 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.736 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:07.737 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:07.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:07.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:07.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:08.220 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:07.757+0000 7fc8936e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:20:08.232 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:20:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:20:07.911+0000 7f3e2f446200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:20:08.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:08.399+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 14) v1 ==== 89903+0+0 (secure 0 0 0) 0x7f661c005630 con 0x7f6624102570 2023-08-25T02:20:08.719 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:08.455+0000 7fc8936e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:20:08.719 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:08.583+0000 7fc8936e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: Standby manager daemon z started 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:20:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]: dispatch 2023-08-25T02:20:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]': finished 2023-08-25T02:20:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: osdmap e5: 1 total, 0 up, 1 in 2023-08-25T02:20:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:08.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:08.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: Standby manager daemon z started 2023-08-25T02:20:08.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:20:08.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:08.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]: dispatch 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]': finished 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: osdmap e5: 1 total, 0 up, 1 in 2023-08-25T02:20:08.739 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:08.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:08.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: Standby manager daemon z started 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1696613554' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:08.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]: dispatch 2023-08-25T02:20:08.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1211817756' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94583d02-1254-4da8-aae0-95aebc0f9804"}]': finished 2023-08-25T02:20:08.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: osdmap e5: 1 total, 0 up, 1 in 2023-08-25T02:20:08.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:09.101 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:08.773+0000 7fc8936e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:20:09.102 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:08.885+0000 7fc8936e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:20:09.414 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:09.100+0000 7fc8936e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:20:09.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:09 smithi122 ceph-mon[100457]: mgrmap e14: x(active, since 65s), standbys: z 2023-08-25T02:20:09.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:20:09.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1045264041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:09.730 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:09 smithi121 ceph-mon[100099]: mgrmap e14: x(active, since 65s), standbys: z 2023-08-25T02:20:09.730 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:20:09.731 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1045264041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:09 smithi089 ceph-mon[98732]: mgrmap e14: x(active, since 65s), standbys: z 2023-08-25T02:20:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:20:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1045264041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:10.222 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:20:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:20:09.949+0000 7fc8936e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:20:10.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:10.417+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 15) v1 ==== 134615+0+0 (secure 0 0 0) 0x7f661c012d10 con 0x7f6624102570 2023-08-25T02:20:10.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: Standby manager daemon y started 2023-08-25T02:20:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:20:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:10.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:20:10.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:10 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:10.746 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:10.746 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: Standby manager daemon y started 2023-08-25T02:20:10.746 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:20:10.746 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:10.746 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:20:10.747 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:10 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:10.841 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: Standby manager daemon y started 2023-08-25T02:20:10.842 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:20:10.842 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:20:10.843 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:20:10.843 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:10 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/1002110488' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:20:11.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:11 smithi121 ceph-mon[100099]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-08-25T02:20:11.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:20:11.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:11 smithi122 ceph-mon[100457]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-08-25T02:20:11.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:20:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:11 smithi089 ceph-mon[98732]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-08-25T02:20:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:20:12.578 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:12 smithi089 ceph-mon[98732]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:12.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:12 smithi121 ceph-mon[100099]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:12.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:12 smithi122 ceph-mon[100457]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:14.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:14 smithi089 ceph-mon[98732]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:14.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:14 smithi121 ceph-mon[100099]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:14.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:14 smithi122 ceph-mon[100457]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:16.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-08-25T02:20:16.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:16.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:16 smithi089 ceph-mon[98732]: Deploying daemon osd.0 on smithi089 2023-08-25T02:20:16.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:16 smithi089 ceph-mon[98732]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-08-25T02:20:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:16 smithi121 ceph-mon[100099]: Deploying daemon osd.0 on smithi089 2023-08-25T02:20:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:16 smithi121 ceph-mon[100099]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-08-25T02:20:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:16 smithi122 ceph-mon[100457]: Deploying daemon osd.0 on smithi089 2023-08-25T02:20:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:16 smithi122 ceph-mon[100457]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:18 smithi089 ceph-mon[98732]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:18 smithi121 ceph-mon[100099]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:18.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:18 smithi122 ceph-mon[100457]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:20.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:20.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:20 smithi089 ceph-mon[98732]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:20 smithi121 ceph-mon[100099]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:20.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:20.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:20.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:20 smithi122 ceph-mon[100457]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:21.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.756+0000 7f66217fa700 1 -- 172.21.15.89:0/2274284897 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f662402cbe0 con 0x7f660c0383e0 2023-08-25T02:20:21.760 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 0 on host 'smithi089' 2023-08-25T02:20:21.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 msgr2=0x7f660c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:21.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 0x7f660c03a8a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6624101a30 tx=0x7f6614007400 comp rx=0 tx=0).stop 2023-08-25T02:20:21.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 msgr2=0x7f662419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:21.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f661c0094c0 tx=0x7f661c0094f0 comp rx=0 tx=0).stop 2023-08-25T02:20:21.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 shutdown_connections 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f660c0383e0 0x7f660c03a8a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66241009d0 0x7f6624196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6624102570 0x7f662419dd20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 --2- 172.21.15.89:0/2274284897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6624101bd0 0x7f6624196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 >> 172.21.15.89:0/2274284897 conn(0x7f66240fbf30 msgr2=0x7f66241050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:20:21.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.758+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 shutdown_connections 2023-08-25T02:20:21.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:20:21.759+0000 7f662a589700 1 -- 172.21.15.89:0/2274284897 wait complete. 2023-08-25T02:20:22.204 DEBUG:teuthology.orchestra.run.smithi089:osd.0> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.0.service 2023-08-25T02:20:22.214 INFO:tasks.cephadm:Deploying osd.1 on smithi121 with /dev/vg_nvme/lv_4... 2023-08-25T02:20:22.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-08-25T02:20:22.245 INFO:journalctl@ceph.osd.0.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:20:22.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:22.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:22.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:22.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:22.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:22.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:22.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:22.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:22.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.604 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.604 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:22.605 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:22.605 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.609 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:22.609 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.609 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:22.609 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:22.794 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:20:23.197 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:20:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:20:23.098+0000 7f66010fa640 -1 osd.0 0 log_to_monitors true 2023-08-25T02:20:23.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:23 smithi121 ceph-mon[100099]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:23.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:23 smithi121 ceph-mon[100099]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:23.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:23 smithi122 ceph-mon[100457]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:23.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:23 smithi122 ceph-mon[100457]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:23.590 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:23 smithi089 ceph-mon[98732]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:23.590 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:23 smithi089 ceph-mon[98732]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-08-25T02:20:24.284 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-08-25T02:20:24.284 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-08-25T02:20:24.285 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2023-08-25T02:20:24.285 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2023-08-25T02:20:24.285 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111319 s, 942 MB/s 2023-08-25T02:20:24.285 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2023-08-25T02:20:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-08-25T02:20:24.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: osdmap e6: 1 total, 0 up, 1 in 2023-08-25T02:20:24.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:24.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:24 smithi121 ceph-mon[100099]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-08-25T02:20:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: osdmap e6: 1 total, 0 up, 1 in 2023-08-25T02:20:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:24 smithi122 ceph-mon[100457]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:24.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:20:24.206+0000 7f65f5f70700 -1 osd.0 0 waiting for initial osdmap 2023-08-25T02:20:24.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:20:24.209+0000 7f65f0562700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:20:24.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-08-25T02:20:24.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: osdmap e6: 1 total, 0 up, 1 in 2023-08-25T02:20:24.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:24.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: from='osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:20:24.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:24 smithi089 ceph-mon[98732]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:24.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2023-08-25T02:20:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:20:25.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: osdmap e7: 1 total, 0 up, 1 in 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi089 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:25.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 17168M 2023-08-25T02:20:25.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:25.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:25.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:25 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:20:25.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: osdmap e7: 1 total, 0 up, 1 in 2023-08-25T02:20:25.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi089 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 17168M 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:25.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:25.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:25 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:20:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: osdmap e7: 1 total, 0 up, 1 in 2023-08-25T02:20:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi089 2023-08-25T02:20:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 17168M 2023-08-25T02:20:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:25 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:26.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:20:26.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:20:26.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] boot 2023-08-25T02:20:26.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: osdmap e8: 1 total, 1 up, 1 in 2023-08-25T02:20:26.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:26.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:26 smithi121 ceph-mon[100099]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:26.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:20:26.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:20:26.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] boot 2023-08-25T02:20:26.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: osdmap e8: 1 total, 1 up, 1 in 2023-08-25T02:20:26.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:26.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:26 smithi122 ceph-mon[100457]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:26.525 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/2643915904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80ff8e0 msgr2=0x7f4db80ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/2643915904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80ff8e0 0x7f4db80ffd00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4da80099c0 tx=0x7f4da8009cd0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/2643915904 shutdown_connections 2023-08-25T02:20:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/2643915904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80ff8e0 0x7f4db80ffd00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/2643915904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8101480 0x7f4db8107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/2643915904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8100ae0 0x7f4db8100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/2643915904 >> 172.21.15.121:0/2643915904 conn(0x7f4db80fae60 msgr2=0x7f4db80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/2643915904 shutdown_connections 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.510+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/2643915904 wait complete. 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 Processor -- start 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 -- start start 2023-08-25T02:20:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80ff8e0 0x7f4db8194500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 0x7f4db819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819c090 con 0x7f4db80ff8e0 2023-08-25T02:20:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819c200 con 0x7f4db8100ae0 2023-08-25T02:20:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4dbd5cf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819c3c0 con 0x7f4db8101480 2023-08-25T02:20:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4db67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4db67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41796/0 (socket says 172.21.15.121:41796) 2023-08-25T02:20:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.511+0000 7f4db67fc700 1 -- 172.21.15.121:0/1975723680 learned_addr learned my addr 172.21.15.121:0/1975723680 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-08-25T02:20:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db6ffd700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80ff8e0 0x7f4db8194500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db77fe700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 0x7f4db819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 -- 172.21.15.121:0/1975723680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 msgr2=0x7f4db819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 0x7f4db819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 -- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80ff8e0 msgr2=0x7f4db8194500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80ff8e0 0x7f4db8194500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 -- 172.21.15.121:0/1975723680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80095f0 con 0x7f4db8100ae0 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db77fe700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 0x7f4db819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4db67fc700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b530 tx=0x7f4dac00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac011830 con 0x7f4db8100ae0 2023-08-25T02:20:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dac011e50 con 0x7f4db8100ae0 2023-08-25T02:20:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db819c6a0 con 0x7f4db8100ae0 2023-08-25T02:20:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac00f2e0 con 0x7f4db8100ae0 2023-08-25T02:20:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.512+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db819cbf0 con 0x7f4db8100ae0 2023-08-25T02:20:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.513+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db810b160 con 0x7f4db8100ae0 2023-08-25T02:20:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.514+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) v1 ==== 134615+0+0 (secure 0 0 0) 0x7f4dac010380 con 0x7f4db8100ae0 2023-08-25T02:20:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.514+0000 7f4d9ffff700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 0x7f4da00a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.515+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f4dac0c9320 con 0x7f4db8100ae0 2023-08-25T02:20:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.515+0000 7f4db6ffd700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 0x7f4da00a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.515+0000 7f4db6ffd700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 0x7f4da00a2480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009f50 tx=0x7f4da800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.519+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4dac094a80 con 0x7f4db8100ae0 2023-08-25T02:20:26.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:20:26.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:20:26.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: osd.0 [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] boot 2023-08-25T02:20:26.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: osdmap e8: 1 total, 1 up, 1 in 2023-08-25T02:20:26.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:20:26.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:26 smithi089 ceph-mon[98732]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-08-25T02:20:26.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:26.711+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db819d1e0 con 0x7f4da009ffc0 2023-08-25T02:20:27.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:27 smithi121 ceph-mon[100099]: osdmap e9: 1 total, 1 up, 1 in 2023-08-25T02:20:27.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:27.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:27.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:27.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:27 smithi122 ceph-mon[100457]: osdmap e9: 1 total, 1 up, 1 in 2023-08-25T02:20:27.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:27.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:27.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:27.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:27 smithi089 ceph-mon[98732]: osdmap e9: 1 total, 1 up, 1 in 2023-08-25T02:20:27.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:27.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:27.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:28.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:28 smithi122 ceph-mon[100457]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:28.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:28 smithi122 ceph-mon[100457]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:28.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:28 smithi089 ceph-mon[98732]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:28.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:28 smithi089 ceph-mon[98732]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:28.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:28 smithi121 ceph-mon[100099]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:28.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:28 smithi121 ceph-mon[100099]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:29.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:29 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/575255812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:29 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:29 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]': finished 2023-08-25T02:20:29.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:29 smithi122 ceph-mon[100457]: osdmap e10: 2 total, 1 up, 2 in 2023-08-25T02:20:29.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:29 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/575255812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:29 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:29 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]': finished 2023-08-25T02:20:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:29 smithi089 ceph-mon[98732]: osdmap e10: 2 total, 1 up, 2 in 2023-08-25T02:20:29.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:29.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:29 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/575255812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:29 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]: dispatch 2023-08-25T02:20:29.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:29 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6112d305-65d5-4bdd-b560-8baba3ca0e98"}]': finished 2023-08-25T02:20:29.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:29 smithi121 ceph-mon[100099]: osdmap e10: 2 total, 1 up, 2 in 2023-08-25T02:20:29.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:30 smithi089 ceph-mon[98732]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:30.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:30 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/760939243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:30.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:30 smithi121 ceph-mon[100099]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:30.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:30 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/760939243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:30.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:30 smithi122 ceph-mon[100457]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:30.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:30 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/760939243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:32 smithi089 ceph-mon[98732]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:32.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:32 smithi121 ceph-mon[100099]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:32.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:32 smithi122 ceph-mon[100457]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:34.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:34 smithi089 ceph-mon[98732]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:34.590 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:34 smithi121 ceph-mon[100099]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:34.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:34 smithi122 ceph-mon[100457]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:36.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:36 smithi121 ceph-mon[100099]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:36.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-08-25T02:20:36.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:36.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:36 smithi089 ceph-mon[98732]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:36.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-08-25T02:20:36.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:36.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:36 smithi122 ceph-mon[100457]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:36.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-08-25T02:20:36.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:37.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:37 smithi121 ceph-mon[100099]: Deploying daemon osd.1 on smithi121 2023-08-25T02:20:37.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:37 smithi089 ceph-mon[98732]: Deploying daemon osd.1 on smithi121 2023-08-25T02:20:37.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:37 smithi122 ceph-mon[100457]: Deploying daemon osd.1 on smithi121 2023-08-25T02:20:38.225 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:38 smithi121 ceph-mon[100099]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:38.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:38 smithi089 ceph-mon[98732]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:38.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:38 smithi122 ceph-mon[100457]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:40.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:40 smithi121 ceph-mon[100099]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:40.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:40 smithi089 ceph-mon[98732]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:40 smithi122 ceph-mon[100457]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:41.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:41.731 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.731 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.731 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:41.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:41.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:42.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:42 smithi121 ceph-mon[100099]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:42.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:42.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:42 smithi122 ceph-mon[100457]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:42 smithi089 ceph-mon[98732]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.279 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.274+0000 7f4d9ffff700 1 -- 172.21.15.121:0/1975723680 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4db819d1e0 con 0x7f4da009ffc0 2023-08-25T02:20:43.280 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 1 on host 'smithi121' 2023-08-25T02:20:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 msgr2=0x7f4da00a2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 0x7f4da00a2480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009f50 tx=0x7f4da800b540 comp rx=0 tx=0).stop 2023-08-25T02:20:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 msgr2=0x7f4db8194a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:43.288 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b530 tx=0x7f4dac00b840 comp rx=0 tx=0).stop 2023-08-25T02:20:43.289 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 shutdown_connections 2023-08-25T02:20:43.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4db8101480 0x7f4db819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:43.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80ff8e0 0x7f4db8194500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:43.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4da009ffc0 0x7f4da00a2480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:43.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 --2- 172.21.15.121:0/1975723680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db8100ae0 0x7f4db8194a40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:43.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 >> 172.21.15.121:0/1975723680 conn(0x7f4db80fae60 msgr2=0x7f4db8068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:20:43.291 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 shutdown_connections 2023-08-25T02:20:43.291 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:20:43.277+0000 7f4dbd5cf700 1 -- 172.21.15.121:0/1975723680 wait complete. 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:43.685 DEBUG:teuthology.orchestra.run.smithi121:osd.1> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.1.service 2023-08-25T02:20:43.697 INFO:tasks.cephadm:Deploying osd.2 on smithi122 with /dev/vg_nvme/lv_4... 2023-08-25T02:20:43.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-08-25T02:20:43.720 INFO:journalctl@ceph.osd.1.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:20:43.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:43.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:43.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:43.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:43.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:43.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.840 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.841 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:43.841 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:20:44.273 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:20:44.468 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:20:44 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:20:44.398+0000 7f019baa9640 -1 osd.1 0 log_to_monitors true 2023-08-25T02:20:44.725 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:44 smithi122 ceph-mon[100457]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:44.726 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:44 smithi122 ceph-mon[100457]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:44.726 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:44 smithi122 ceph-mon[100457]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:44 smithi089 ceph-mon[98732]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:44 smithi089 ceph-mon[98732]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:44 smithi089 ceph-mon[98732]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:44 smithi121 ceph-mon[100099]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:44 smithi121 ceph-mon[100099]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:44 smithi121 ceph-mon[100099]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-08-25T02:20:45.946 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-08-25T02:20:45.946 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-08-25T02:20:45.947 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-08-25T02:20:45.947 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-08-25T02:20:45.947 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110433 s, 950 MB/s 2023-08-25T02:20:45.947 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-08-25T02:20:46.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2023-08-25T02:20:46.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-08-25T02:20:46.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: osdmap e11: 2 total, 1 up, 2 in 2023-08-25T02:20:46.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:46.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:46.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:46.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-08-25T02:20:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: osdmap e11: 2 total, 1 up, 2 in 2023-08-25T02:20:46.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:46.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:46.747 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.747 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.747 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:46.748 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.749 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:46.749 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:46.750 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.768 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:20:46.247+0000 7f019091f700 -1 osd.1 0 waiting for initial osdmap 2023-08-25T02:20:46.769 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:20:46 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:20:46.253+0000 7f0188f0d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:20:46.770 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-08-25T02:20:46.770 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: osdmap e11: 2 total, 1 up, 2 in 2023-08-25T02:20:46.770 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:46.770 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.772 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:20:46.772 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:20:46.773 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:20:46.916 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:20:47.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi121 2023-08-25T02:20:47.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 17168M 2023-08-25T02:20:47.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:20:47.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: osdmap e12: 2 total, 1 up, 2 in 2023-08-25T02:20:47.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:47.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:47 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:47.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi121 2023-08-25T02:20:47.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 17168M 2023-08-25T02:20:47.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:20:47.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: osdmap e12: 2 total, 1 up, 2 in 2023-08-25T02:20:47.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:47.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:47 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:47.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi121 2023-08-25T02:20:47.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 17168M 2023-08-25T02:20:47.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:20:47.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: osdmap e12: 2 total, 1 up, 2 in 2023-08-25T02:20:47.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:47.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3712025153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf8101450 msgr2=0x7fcbf8107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3712025153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf8101450 0x7fcbf8107d00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000b0a0 tx=0x7fcbf000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3712025153 shutdown_connections 2023-08-25T02:20:48.235 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3712025153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf80ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3712025153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf8101450 0x7fcbf8107d00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3712025153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 0x7fcbf8100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3712025153 >> 172.21.15.122:0/3712025153 conn(0x7fcbf80fae30 msgr2=0x7fcbf80fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3712025153 shutdown_connections 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3712025153 wait complete. 2023-08-25T02:20:48.236 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 Processor -- start 2023-08-25T02:20:48.237 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.232+0000 7fcbfe9f7700 1 -- start start 2023-08-25T02:20:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 0x7fcbf819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:48.238 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8101450 0x7fcbf819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf819e2e0 con 0x7fcbf8100ab0 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf819e450 con 0x7fcbf8101450 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf819e610 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33064/0 (socket says 172.21.15.122:33064) 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 -- 172.21.15.122:0/3567762002 learned_addr learned my addr 172.21.15.122:0/3567762002 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:20:48.239 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf77fe700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 0x7fcbf819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfcf94700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8101450 0x7fcbf819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 -- 172.21.15.122:0/3567762002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8101450 msgr2=0x7fcbf819db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8101450 0x7fcbf819db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 -- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 msgr2=0x7fcbf819b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 0x7fcbf819b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:20:48.240 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 -- 172.21.15.122:0/3567762002 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf0009d00 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf7fff700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00b500 tx=0x7fcbec00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec011830 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf819e8f0 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbec011e50 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec010dc0 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.241 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.233+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf819ee70 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.242 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.234+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8106e10 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.235+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 15) v1 ==== 134615+0+0 (secure 0 0 0) 0x7fcbec010380 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.236+0000 7fcbf57fa700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 0x7fcbe00a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:20:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.236+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fcbec0c93f0 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.236+0000 7fcbf77fe700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 0x7fcbe00a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:20:48.243 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.236+0000 7fcbf77fe700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 0x7fcbe00a2380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80694c0 tx=0x7fcbe8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:20:48.246 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.240+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fcbec0949d0 con 0x7fcbf80ff8b0 2023-08-25T02:20:48.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:20:48.436+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf802ceb0 con 0x7fcbe009fec0 2023-08-25T02:20:48.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:20:48.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:20:48.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:48.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] boot 2023-08-25T02:20:48.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: osdmap e13: 2 total, 2 up, 2 in 2023-08-25T02:20:48.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:48.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:20:48.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:20:48.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:48.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] boot 2023-08-25T02:20:48.591 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: osdmap e13: 2 total, 2 up, 2 in 2023-08-25T02:20:48.591 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:48.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:20:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:20:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-08-25T02:20:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: osd.1 [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] boot 2023-08-25T02:20:48.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: osdmap e13: 2 total, 2 up, 2 in 2023-08-25T02:20:48.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:20:49.562 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:49 smithi122 ceph-mon[100457]: osdmap e14: 2 total, 2 up, 2 in 2023-08-25T02:20:49.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:49 smithi122 ceph-mon[100457]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:49.571 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:49 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:49.581 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:49 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:49.585 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:49 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:49.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:49 smithi089 ceph-mon[98732]: osdmap e14: 2 total, 2 up, 2 in 2023-08-25T02:20:49.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:49 smithi089 ceph-mon[98732]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:49.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:49 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:49.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:49 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:49.586 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:49 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:49.733 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:49 smithi121 ceph-mon[100099]: osdmap e14: 2 total, 2 up, 2 in 2023-08-25T02:20:49.734 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:49 smithi121 ceph-mon[100099]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:20:49.734 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:49 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:20:49.734 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:49 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:20:49.734 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:49 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:50 smithi122 ceph-mon[100457]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:50.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:50 smithi089 ceph-mon[98732]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:50.726 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:50 smithi121 ceph-mon[100099]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:51 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/799044494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:51 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:51 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]': finished 2023-08-25T02:20:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:51 smithi089 ceph-mon[98732]: osdmap e15: 3 total, 2 up, 3 in 2023-08-25T02:20:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:20:51.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:51 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/799044494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:51 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:51 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]': finished 2023-08-25T02:20:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:51 smithi121 ceph-mon[100099]: osdmap e15: 3 total, 2 up, 3 in 2023-08-25T02:20:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:20:51.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:51 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/799044494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:51 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]: dispatch 2023-08-25T02:20:51.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:51 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8a1640c-fce8-4ac8-a53f-857213701817"}]': finished 2023-08-25T02:20:51.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:51 smithi122 ceph-mon[100457]: osdmap e15: 3 total, 2 up, 3 in 2023-08-25T02:20:51.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:20:52.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:52 smithi089 ceph-mon[98732]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:52.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:52 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/849682570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:52.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:52 smithi121 ceph-mon[100099]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:52.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:52 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/849682570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:52.735 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:52 smithi122 ceph-mon[100457]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:52.736 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:52 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/849682570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:20:54.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:54 smithi089 ceph-mon[98732]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:54.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:54 smithi121 ceph-mon[100099]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:54.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:54 smithi122 ceph-mon[100457]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:56.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:56 smithi089 ceph-mon[98732]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:56.377 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:56 smithi122 ceph-mon[100457]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:56.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:56 smithi121 ceph-mon[100099]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:58 smithi122 ceph-mon[100457]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-08-25T02:20:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:58.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:58 smithi089 ceph-mon[98732]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:58.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-08-25T02:20:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:58.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:58 smithi121 ceph-mon[100099]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:20:58.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-08-25T02:20:58.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:20:59.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:20:59 smithi089 ceph-mon[98732]: Deploying daemon osd.2 on smithi122 2023-08-25T02:20:59.586 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:20:59 smithi122 ceph-mon[100457]: Deploying daemon osd.2 on smithi122 2023-08-25T02:20:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:20:59 smithi121 ceph-mon[100099]: Deploying daemon osd.2 on smithi122 2023-08-25T02:21:00.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:00 smithi089 ceph-mon[98732]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:00.620 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:00 smithi122 ceph-mon[100457]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:00.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:00 smithi121 ceph-mon[100099]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:02.589 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:02 smithi089 ceph-mon[98732]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:02.638 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:02 smithi122 ceph-mon[100457]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:02.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:02 smithi121 ceph-mon[100099]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:03.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:03.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:03.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:03.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:03 smithi122 ceph-mon[100457]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:04.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:04.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:04.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:04.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:03 smithi089 ceph-mon[98732]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:04.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:04.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:04.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:04.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:03 smithi121 ceph-mon[100099]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:05.849 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.848+0000 7fcbf57fa700 1 -- 172.21.15.122:0/3567762002 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcbf802ceb0 con 0x7fcbe009fec0 2023-08-25T02:21:05.850 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 2 on host 'smithi122' 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 msgr2=0x7fcbe00a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 0x7fcbe00a2380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80694c0 tx=0x7fcbe8009260 comp rx=0 tx=0).stop 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 msgr2=0x7fcbf8068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00b500 tx=0x7fcbec00b810 comp rx=0 tx=0).stop 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 shutdown_connections 2023-08-25T02:21:05.853 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcbe009fec0 0x7fcbe00a2380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8101450 0x7fcbf819db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80ff8b0 0x7fcbf8068f60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 --2- 172.21.15.122:0/3567762002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbf8100ab0 0x7fcbf819b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 >> 172.21.15.122:0/3567762002 conn(0x7fcbf80fae30 msgr2=0x7fcbf8103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 shutdown_connections 2023-08-25T02:21:05.854 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:05.850+0000 7fcbfe9f7700 1 -- 172.21.15.122:0/3567762002 wait complete. 2023-08-25T02:21:06.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:06.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:06.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:06.232 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:06.232 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.232 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:06.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:06.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:06.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:06.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:06.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:06.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:06.413 DEBUG:teuthology.orchestra.run.smithi122:osd.2> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.2.service 2023-08-25T02:21:06.417 INFO:tasks.cephadm:Deploying osd.3 on smithi089 with /dev/vg_nvme/lv_3... 2023-08-25T02:21:06.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-08-25T02:21:06.452 INFO:journalctl@ceph.osd.2.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:21:06.937 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:21:06.673+0000 7f48ceb2e640 -1 osd.2 0 log_to_monitors true 2023-08-25T02:21:06.986 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:21:07.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:07 smithi122 ceph-mon[100457]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:07.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:07 smithi122 ceph-mon[100457]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:07.284 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:07 smithi089 ceph-mon[98732]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:07.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:07 smithi089 ceph-mon[98732]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:07.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:07 smithi121 ceph-mon[100099]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:07.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:07 smithi121 ceph-mon[100099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-08-25T02:21:08.053 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:21:08.046+0000 7f48c39a4700 -1 osd.2 0 waiting for initial osdmap 2023-08-25T02:21:08.053 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-08-25T02:21:08.053 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: osdmap e16: 3 total, 2 up, 3 in 2023-08-25T02:21:08.053 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:08.054 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.054 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.054 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-mon[100457]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:08.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-08-25T02:21:08.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: osdmap e16: 3 total, 2 up, 3 in 2023-08-25T02:21:08.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:08.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.336 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:08 smithi089 ceph-mon[98732]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:08.386 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:21:08.050+0000 7f48be797700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:21:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-08-25T02:21:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-08-25T02:21:08.450 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-08-25T02:21:08.450 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-08-25T02:21:08.450 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110224 s, 951 MB/s 2023-08-25T02:21:08.450 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-08-25T02:21:08.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-08-25T02:21:08.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: osdmap e16: 3 total, 2 up, 3 in 2023-08-25T02:21:08.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:08.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: from='osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:21:08.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:08 smithi121 ceph-mon[100099]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:08.842 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2023-08-25T02:21:09.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:21:09.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: osdmap e17: 3 total, 2 up, 3 in 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:09.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.235 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:09.235 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:09.235 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:21:09.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: osdmap e17: 3 total, 2 up, 3 in 2023-08-25T02:21:09.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:09.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:09.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:09.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.402 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: osdmap e17: 3 total, 2 up, 3 in 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:10.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:21:10.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:21:10.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi122 2023-08-25T02:21:10.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 17168M 2023-08-25T02:21:10.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] boot 2023-08-25T02:21:10.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: osdmap e18: 3 total, 3 up, 3 in 2023-08-25T02:21:10.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:10.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:10.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:10.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:21:10.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:21:10.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi122 2023-08-25T02:21:10.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 17168M 2023-08-25T02:21:10.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] boot 2023-08-25T02:21:10.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: osdmap e18: 3 total, 3 up, 3 in 2023-08-25T02:21:10.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:10.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:10.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:10.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:21:10.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:21:10.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi122 2023-08-25T02:21:10.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 17168M 2023-08-25T02:21:10.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: osd.2 [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] boot 2023-08-25T02:21:10.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: osdmap e18: 3 total, 3 up, 3 in 2023-08-25T02:21:10.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:21:10.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-08-25T02:21:10.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:10.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 -- 172.21.15.89:0/3524162991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d880ffb00 msgr2=0x7f5d880fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:10.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/3524162991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d880ffb00 0x7f5d880fff80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009990 tx=0x7f5d78009ca0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 -- 172.21.15.89:0/3524162991 shutdown_connections 2023-08-25T02:21:10.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/3524162991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d88100f30 0x7f5d88101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/3524162991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d881004c0 0x7f5d88107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/3524162991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d880ffb00 0x7f5d880fff80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 -- 172.21.15.89:0/3524162991 >> 172.21.15.89:0/3524162991 conn(0x7f5d880fa5d0 msgr2=0x7f5d880fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:10.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 -- 172.21.15.89:0/3524162991 shutdown_connections 2023-08-25T02:21:10.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.573+0000 7f5d8ce13700 1 -- 172.21.15.89:0/3524162991 wait complete. 2023-08-25T02:21:10.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 Processor -- start 2023-08-25T02:21:10.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 -- start start 2023-08-25T02:21:10.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 0x7f5d88195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:10.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d881004c0 0x7f5d88195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:10.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:10.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819d100 con 0x7f5d88100f30 2023-08-25T02:21:10.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819d240 con 0x7f5d881004c0 2023-08-25T02:21:10.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8ce13700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819d3d0 con 0x7f5d880ffb00 2023-08-25T02:21:10.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d86d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d86d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45856/0 (socket says 172.21.15.89:45856) 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d86d9d700 1 -- 172.21.15.89:0/2036431158 learned_addr learned my addr 172.21.15.89:0/2036431158 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d85d9b700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d881004c0 0x7f5d88195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.574+0000 7f5d8659c700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 0x7f5d88195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 -- 172.21.15.89:0/2036431158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 msgr2=0x7f5d88195570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 0x7f5d88195570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 -- 172.21.15.89:0/2036431158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d881004c0 msgr2=0x7f5d88195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d881004c0 0x7f5d88195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 -- 172.21.15.89:0/2036431158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780095f0 con 0x7f5d88100f30 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d8659c700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 0x7f5d88195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d86d9d700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000add0 tx=0x7f5d8000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8000f070 con 0x7f5d88100f30 2023-08-25T02:21:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d80007b30 con 0x7f5d88100f30 2023-08-25T02:21:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8819d6b0 con 0x7f5d88100f30 2023-08-25T02:21:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80005310 con 0x7f5d88100f30 2023-08-25T02:21:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.575+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8819dbb0 con 0x7f5d88100f30 2023-08-25T02:21:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.576+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8810b0b0 con 0x7f5d88100f30 2023-08-25T02:21:10.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.577+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 134615+0+0 (secure 0 0 0) 0x7f5d8000c940 con 0x7f5d88100f30 2023-08-25T02:21:10.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.577+0000 7f5d777fe700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 0x7f5d700a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:10.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.578+0000 7f5d8659c700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 0x7f5d700a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:10.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.578+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f5d800c98b0 con 0x7f5d88100f30 2023-08-25T02:21:10.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.578+0000 7f5d8659c700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 0x7f5d700a2400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005fd0 tx=0x7f5d7c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:10.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.582+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5d80094b90 con 0x7f5d88100f30 2023-08-25T02:21:10.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:10.777+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d88060d60 con 0x7f5d7009ff40 2023-08-25T02:21:11.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:11.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: osdmap e19: 3 total, 3 up, 3 in 2023-08-25T02:21:11.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:11.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:11.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:11.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:11 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:11.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:11.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: osdmap e19: 3 total, 3 up, 3 in 2023-08-25T02:21:11.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:11.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:11.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:11.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:11 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:11.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:11.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: osdmap e19: 3 total, 3 up, 3 in 2023-08-25T02:21:11.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:21:11.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:11.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:11.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:11 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:11.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114771]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114771]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114771]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114771]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114812]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:11.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114812]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:12.069 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:21:11 smithi089 sudo[114812]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:12.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:12 smithi089 ceph-mon[98732]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:12.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:12.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:12 smithi089 ceph-mon[98732]: osdmap e20: 3 total, 3 up, 3 in 2023-08-25T02:21:12.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:12 smithi089 ceph-mon[98732]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:12.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:12.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:12 smithi121 ceph-mon[100099]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:12.345 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:12.345 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:12 smithi121 ceph-mon[100099]: osdmap e20: 3 total, 3 up, 3 in 2023-08-25T02:21:12.345 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:12 smithi121 ceph-mon[100099]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:12.345 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:12.345 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107420]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:12.346 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:12.346 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107420]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:12.346 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107420]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:12 smithi122 ceph-mon[100457]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:21:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:12 smithi122 ceph-mon[100457]: osdmap e20: 3 total, 3 up, 3 in 2023-08-25T02:21:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:12 smithi122 ceph-mon[100457]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:12.724 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107423]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:12.724 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:12.724 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107423]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:12.725 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:21:12 smithi121 sudo[107423]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:12.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:12.858+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5d80094f60 con 0x7f5d88100f30 2023-08-25T02:21:12.869 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108406]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:12.869 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108406]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:12.870 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108406]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:12.870 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108406]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:13.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108409]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:13.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:13.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108409]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:13.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:21:12 smithi122 sudo[108409]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:13.397 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114909]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:13.398 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:13.398 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114909]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:13.398 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114909]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:13.663 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 ceph-mon[100457]: osdmap e21: 3 total, 3 up, 3 in 2023-08-25T02:21:13.664 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.664 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 ceph-mon[100457]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-08-25T02:21:13.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108412]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:13.667 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:13.668 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108412]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:13.669 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108412]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:13.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:13 smithi121 ceph-mon[100099]: osdmap e21: 3 total, 3 up, 3 in 2023-08-25T02:21:13.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:13 smithi121 ceph-mon[100099]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-08-25T02:21:13.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114912]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114912]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114912]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 sudo[114912]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 ceph-mon[98732]: osdmap e21: 3 total, 3 up, 3 in 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 ceph-mon[98732]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-08-25T02:21:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:13.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108415]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:13.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:13.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108415]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:13.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:13 smithi122 sudo[108415]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:14.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107426]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-08-25T02:21:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107426]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107426]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:14.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107429]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-08-25T02:21:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-08-25T02:21:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107429]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-08-25T02:21:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 sudo[107429]: pam_unix(sudo:session): session closed for user root 2023-08-25T02:21:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]: dispatch 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]': finished 2023-08-25T02:21:14.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: osdmap e22: 4 total, 3 up, 4 in 2023-08-25T02:21:14.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/838370688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:14.725 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:14 smithi121 ceph-mon[100099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:14.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]: dispatch 2023-08-25T02:21:14.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]': finished 2023-08-25T02:21:14.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: osdmap e22: 4 total, 3 up, 4 in 2023-08-25T02:21:14.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.735 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/838370688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:14.735 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:14 smithi122 ceph-mon[100457]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:14.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1620985029' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff"}]': finished 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: osdmap e22: 4 total, 3 up, 4 in 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:21:14.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:21:14.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:21:14.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/838370688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:14.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:14 smithi089 ceph-mon[98732]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-08-25T02:21:16.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:16 smithi089 ceph-mon[98732]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:16.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:16 smithi121 ceph-mon[100099]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:16.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:16 smithi122 ceph-mon[100457]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:18.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:18 smithi089 ceph-mon[98732]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:18.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:18 smithi121 ceph-mon[100099]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:18 smithi122 ceph-mon[100457]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:20.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:20 smithi089 ceph-mon[98732]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-08-25T02:21:20.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-08-25T02:21:20.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:20 smithi121 ceph-mon[100099]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-08-25T02:21:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-08-25T02:21:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:20 smithi122 ceph-mon[100457]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-08-25T02:21:20.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-08-25T02:21:20.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:21 smithi089 ceph-mon[98732]: Deploying daemon osd.3 on smithi089 2023-08-25T02:21:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:21 smithi121 ceph-mon[100099]: Deploying daemon osd.3 on smithi089 2023-08-25T02:21:21.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:21 smithi122 ceph-mon[100457]: Deploying daemon osd.3 on smithi089 2023-08-25T02:21:22.522 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:22 smithi089 ceph-mon[98732]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-08-25T02:21:22.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:22 smithi121 ceph-mon[100099]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-08-25T02:21:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:22 smithi122 ceph-mon[100457]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-08-25T02:21:24.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:24 smithi089 ceph-mon[98732]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-08-25T02:21:24.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:24.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:24.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:24.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:24 smithi121 ceph-mon[100099]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-08-25T02:21:24.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:24.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:24.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:24.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:24 smithi122 ceph-mon[100457]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-08-25T02:21:24.740 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:24.740 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:24.740 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:26.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:26 smithi089 ceph-mon[98732]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-08-25T02:21:26.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:26 smithi121 ceph-mon[100099]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-08-25T02:21:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:26 smithi122 ceph-mon[100457]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-08-25T02:21:26.830 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 3 on host 'smithi089' 2023-08-25T02:21:26.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.827+0000 7f5d777fe700 1 -- 172.21.15.89:0/2036431158 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5d88060d60 con 0x7f5d7009ff40 2023-08-25T02:21:26.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 msgr2=0x7f5d700a2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:26.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 0x7f5d700a2400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005fd0 tx=0x7f5d7c005ee0 comp rx=0 tx=0).stop 2023-08-25T02:21:26.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 msgr2=0x7f5d8819cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:26.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000add0 tx=0x7f5d8000c330 comp rx=0 tx=0).stop 2023-08-25T02:21:26.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 shutdown_connections 2023-08-25T02:21:26.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88100f30 0x7f5d8819cb30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d880ffb00 0x7f5d88195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5d7009ff40 0x7f5d700a2400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 --2- 172.21.15.89:0/2036431158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d881004c0 0x7f5d88195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 >> 172.21.15.89:0/2036431158 conn(0x7f5d880fa5d0 msgr2=0x7f5d880fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 shutdown_connections 2023-08-25T02:21:26.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:21:26.830+0000 7f5d8ce13700 1 -- 172.21.15.89:0/2036431158 wait complete. 2023-08-25T02:21:27.279 DEBUG:teuthology.orchestra.run.smithi089:osd.3> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.3.service 2023-08-25T02:21:27.294 INFO:tasks.cephadm:Deploying osd.4 on smithi121 with /dev/vg_nvme/lv_3... 2023-08-25T02:21:27.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-08-25T02:21:27.323 INFO:journalctl@ceph.osd.3.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:21:27.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:27 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:27 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.881 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:21:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:27 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:28.501 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:21:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:21:28.238+0000 7f0484d50640 -1 osd.3 0 log_to_monitors true 2023-08-25T02:21:28.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:28 smithi121 ceph-mon[100099]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-08-25T02:21:28.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:28 smithi121 ceph-mon[100099]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-08-25T02:21:28.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:28 smithi089 ceph-mon[98732]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-08-25T02:21:28.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:28 smithi089 ceph-mon[98732]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-08-25T02:21:28.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:28 smithi122 ceph-mon[100457]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-08-25T02:21:28.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:28 smithi122 ceph-mon[100457]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-08-25T02:21:29.358 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-08-25T02:21:29.358 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-08-25T02:21:29.358 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2023-08-25T02:21:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2023-08-25T02:21:29.359 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109844 s, 955 MB/s 2023-08-25T02:21:29.359 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2023-08-25T02:21:29.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2023-08-25T02:21:30.081 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:21:29.810+0000 7f0479bc6700 -1 osd.3 0 waiting for initial osdmap 2023-08-25T02:21:30.082 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:21:29.816+0000 7f04739b7700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:21:30.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-08-25T02:21:30.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: osdmap e23: 4 total, 3 up, 4 in 2023-08-25T02:21:30.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:30.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:21:30.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi089 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 8584M 2023-08-25T02:21:30.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:30.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.085 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.085 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-08-25T02:21:30.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: osdmap e23: 4 total, 3 up, 4 in 2023-08-25T02:21:30.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:30.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:21:30.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi089 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 8584M 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:30.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-08-25T02:21:30.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: osdmap e23: 4 total, 3 up, 4 in 2023-08-25T02:21:30.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:30.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:21:30.257 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi089 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:30.258 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 8584M 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:30.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:30.260 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:30.260 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:30.297 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:21:31.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:21:31.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: osdmap e24: 4 total, 3 up, 4 in 2023-08-25T02:21:31.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] boot 2023-08-25T02:21:31.089 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: osdmap e25: 4 total, 4 up, 4 in 2023-08-25T02:21:31.089 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:30 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:21:31.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: osdmap e24: 4 total, 3 up, 4 in 2023-08-25T02:21:31.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] boot 2023-08-25T02:21:31.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: osdmap e25: 4 total, 4 up, 4 in 2023-08-25T02:21:31.221 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:30 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.233 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: from='osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:21:31.234 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: osdmap e24: 4 total, 3 up, 4 in 2023-08-25T02:21:31.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: osd.3 [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] boot 2023-08-25T02:21:31.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: osdmap e25: 4 total, 4 up, 4 in 2023-08-25T02:21:31.263 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:30 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:21:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 -- 172.21.15.121:0/2057471774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 msgr2=0x7f4c040ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/2057471774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040ffcd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00b0d0 tx=0x7f4bfc00b3e0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 -- 172.21.15.121:0/2057471774 shutdown_connections 2023-08-25T02:21:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/2057471774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040ffcd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/2057471774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04101450 0x7f4c04107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/2057471774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c04100ab0 0x7f4c04100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 -- 172.21.15.121:0/2057471774 >> 172.21.15.121:0/2057471774 conn(0x7f4c040fae30 msgr2=0x7f4c040fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 -- 172.21.15.121:0/2057471774 shutdown_connections 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.471+0000 7f4c08ad3700 1 -- 172.21.15.121:0/2057471774 wait complete. 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 Processor -- start 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 -- start start 2023-08-25T02:21:31.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 0x7f4c0419b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c04101450 0x7f4c0419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:31.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0419e290 con 0x7f4c04100ab0 2023-08-25T02:21:31.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0419e400 con 0x7f4c040ff8b0 2023-08-25T02:21:31.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c08ad3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0419e5c0 con 0x7f4c04101450 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41872/0 (socket says 172.21.15.121:41872) 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 -- 172.21.15.121:0/769066121 learned_addr learned my addr 172.21.15.121:0/769066121 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c01d9b700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 0x7f4c0419b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 -- 172.21.15.121:0/769066121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c04101450 msgr2=0x7f4c0419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c04101450 0x7f4c0419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 -- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 msgr2=0x7f4c0419b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 0x7f4c0419b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 -- 172.21.15.121:0/769066121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc009d00 con 0x7f4c040ff8b0 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c01d9b700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 0x7f4c0419b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.472+0000 7f4c0259c700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc000f80 tx=0x7f4bfc009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.473+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00e040 con 0x7f4c040ff8b0 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.473+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0419e840 con 0x7f4c040ff8b0 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.473+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bfc01d480 con 0x7f4c040ff8b0 2023-08-25T02:21:31.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.473+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc01b6b0 con 0x7f4c040ff8b0 2023-08-25T02:21:31.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.473+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0419ecb0 con 0x7f4c040ff8b0 2023-08-25T02:21:31.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.474+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c04106e60 con 0x7f4c040ff8b0 2023-08-25T02:21:31.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.475+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f4bfc01d5f0 con 0x7f4c040ff8b0 2023-08-25T02:21:31.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.475+0000 7f4bf37fe700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 0x7f4bec0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:31.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.475+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f4bfc0cbeb0 con 0x7f4c040ff8b0 2023-08-25T02:21:31.498 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.476+0000 7f4c01d9b700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 0x7f4bec0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:31.498 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.476+0000 7f4c01d9b700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 0x7f4bec0a2410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8005fd0 tx=0x7f4bf8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:31.498 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.480+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4bfc096ed0 con 0x7f4c040ff8b0 2023-08-25T02:21:31.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:31.675+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c0402ced0 con 0x7f4bec09ff50 2023-08-25T02:21:32.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: osdmap e26: 4 total, 4 up, 4 in 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:32.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:32 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:32.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:21:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:21:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: osdmap e26: 4 total, 4 up, 4 in 2023-08-25T02:21:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:32.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:32 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:32.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:21:32.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:21:32.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: osdmap e26: 4 total, 4 up, 4 in 2023-08-25T02:21:32.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:32.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:32.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:32.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:32 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:33.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:33 smithi121 ceph-mon[100099]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:33.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:33 smithi121 ceph-mon[100099]: osdmap e27: 4 total, 4 up, 4 in 2023-08-25T02:21:33.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:33 smithi089 ceph-mon[98732]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:33.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:33 smithi089 ceph-mon[98732]: osdmap e27: 4 total, 4 up, 4 in 2023-08-25T02:21:33.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:33 smithi122 ceph-mon[100457]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:33.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:33 smithi122 ceph-mon[100457]: osdmap e27: 4 total, 4 up, 4 in 2023-08-25T02:21:34.334 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: osdmap e28: 4 total, 4 up, 4 in 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/644185497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]': finished 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: osdmap e29: 5 total, 4 up, 5 in 2023-08-25T02:21:34.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:34.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: osdmap e28: 4 total, 4 up, 4 in 2023-08-25T02:21:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-08-25T02:21:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/644185497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]': finished 2023-08-25T02:21:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: osdmap e29: 5 total, 4 up, 5 in 2023-08-25T02:21:34.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: osdmap e28: 4 total, 4 up, 4 in 2023-08-25T02:21:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-08-25T02:21:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/644185497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]: dispatch 2023-08-25T02:21:34.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57ae45d-1fb5-41e6-9c04-ba9c235814af"}]': finished 2023-08-25T02:21:34.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: osdmap e29: 5 total, 4 up, 5 in 2023-08-25T02:21:34.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:35.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:35 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/2630470456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:35.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:35 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/2630470456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:35.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:35 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/2630470456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:36 smithi089 ceph-mon[98732]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:21:36.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:36 smithi121 ceph-mon[100099]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:21:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:36 smithi122 ceph-mon[100457]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:21:38.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:38 smithi089 ceph-mon[98732]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:21:38.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:38 smithi121 ceph-mon[100099]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:21:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:38 smithi122 ceph-mon[100457]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:21:40.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:40 smithi089 ceph-mon[98732]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:21:40.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:40 smithi121 ceph-mon[100099]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:21:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:40 smithi122 ceph-mon[100457]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:21:41.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-08-25T02:21:41.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:41.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-08-25T02:21:41.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:41.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-08-25T02:21:41.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:42.406 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:42 smithi121 ceph-mon[100099]: Deploying daemon osd.4 on smithi121 2023-08-25T02:21:42.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:42 smithi121 ceph-mon[100099]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:42.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:42 smithi089 ceph-mon[98732]: Deploying daemon osd.4 on smithi121 2023-08-25T02:21:42.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:42 smithi089 ceph-mon[98732]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:42 smithi122 ceph-mon[100457]: Deploying daemon osd.4 on smithi121 2023-08-25T02:21:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:42 smithi122 ceph-mon[100457]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:44.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:44 smithi121 ceph-mon[100099]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:44.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:44 smithi089 ceph-mon[98732]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:44.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:44 smithi122 ceph-mon[100457]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:46.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:46 smithi121 ceph-mon[100099]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:46.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:46.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:46 smithi122 ceph-mon[100457]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:46.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:46.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:46 smithi089 ceph-mon[98732]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:47.987 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.985+0000 7f4bf37fe700 1 -- 172.21.15.121:0/769066121 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4c0402ced0 con 0x7f4bec09ff50 2023-08-25T02:21:47.988 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 4 on host 'smithi121' 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 msgr2=0x7f4bec0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 0x7f4bec0a2410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8005fd0 tx=0x7f4bf8005e20 comp rx=0 tx=0).stop 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 msgr2=0x7f4c040697f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc000f80 tx=0x7f4bfc009570 comp rx=0 tx=0).stop 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 shutdown_connections 2023-08-25T02:21:47.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4bec09ff50 0x7f4bec0a2410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c040ff8b0 0x7f4c040697f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c04101450 0x7f4c0419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 --2- 172.21.15.121:0/769066121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100ab0 0x7f4c0419b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 >> 172.21.15.121:0/769066121 conn(0x7f4c040fae30 msgr2=0x7f4c04103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 shutdown_connections 2023-08-25T02:21:47.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:21:47.988+0000 7f4c08ad3700 1 -- 172.21.15.121:0/769066121 wait complete. 2023-08-25T02:21:48.377 DEBUG:teuthology.orchestra.run.smithi121:osd.4> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.4.service 2023-08-25T02:21:48.383 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_3... 2023-08-25T02:21:48.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-08-25T02:21:48.411 INFO:journalctl@ceph.osd.4.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:21:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:48 smithi122 ceph-mon[100457]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:48.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.955 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:21:48.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:48 smithi121 ceph-mon[100099]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:48.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.963 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:48.964 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:49.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:48 smithi089 ceph-mon[98732]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:49.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:49.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:49.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:49.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:49.481 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:21:49 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:21:49.326+0000 7f887c29e640 -1 osd.4 0 log_to_monitors true 2023-08-25T02:21:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:49 smithi121 ceph-mon[100099]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:49.763 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:49 smithi121 ceph-mon[100099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:49.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:49 smithi122 ceph-mon[100457]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:49.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:49 smithi122 ceph-mon[100457]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:49 smithi089 ceph-mon[98732]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:49 smithi089 ceph-mon[98732]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-08-25T02:21:50.655 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:50.655 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-08-25T02:21:50.655 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: osdmap e30: 5 total, 4 up, 5 in 2023-08-25T02:21:50.656 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:50.656 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:50.656 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-mon[100099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:50.696 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-08-25T02:21:50.697 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-08-25T02:21:50.697 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-08-25T02:21:50.697 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-08-25T02:21:50.697 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0106983 s, 980 MB/s 2023-08-25T02:21:50.698 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-08-25T02:21:50.905 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:50.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-08-25T02:21:50.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: osdmap e30: 5 total, 4 up, 5 in 2023-08-25T02:21:50.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:50.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:50.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:50 smithi122 ceph-mon[100457]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:50.968 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:21:50.655+0000 7f8871114700 -1 osd.4 0 waiting for initial osdmap 2023-08-25T02:21:50.968 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:21:50 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:21:50.660+0000 7f886d70a700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:21:51.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:51.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-08-25T02:21:51.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: osdmap e30: 5 total, 4 up, 5 in 2023-08-25T02:21:51.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:51.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: from='osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:51.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:50 smithi089 ceph-mon[98732]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:21:51.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2023-08-25T02:21:51.769 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:21:51.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:21:51.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: osdmap e31: 5 total, 4 up, 5 in 2023-08-25T02:21:51.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:51.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:51.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi121 2023-08-25T02:21:51.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:51.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:51.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:51.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:51.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 8584M 2023-08-25T02:21:51.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:51.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:51.982 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:51.982 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:51.982 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:51.982 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:51.983 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:51.983 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:51.983 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:51 smithi121 ceph-mon[100099]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:52.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:21:52.008 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: osdmap e31: 5 total, 4 up, 5 in 2023-08-25T02:21:52.008 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:52.020 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:52.020 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi121 2023-08-25T02:21:52.021 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.021 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.021 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:52.022 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:52.022 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 8584M 2023-08-25T02:21:52.022 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:52.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:52.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:52.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:52.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:52.028 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.029 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:51 smithi122 ceph-mon[100457]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:52.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:21:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: osdmap e31: 5 total, 4 up, 5 in 2023-08-25T02:21:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:52.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:52.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi121 2023-08-25T02:21:52.089 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.090 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.090 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 8584M 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:52.105 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.106 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:21:52.106 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:52.106 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:21:52.106 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:21:52.106 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:51 smithi089 ceph-mon[98732]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-08-25T02:21:52.895 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:52 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:21:52.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:52 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:21:52.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:52 smithi122 ceph-mon[100457]: osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] boot 2023-08-25T02:21:52.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:52 smithi122 ceph-mon[100457]: osdmap e32: 5 total, 5 up, 5 in 2023-08-25T02:21:52.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:52.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:52 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:21:52.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:52 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:21:52.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:52 smithi121 ceph-mon[100099]: osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] boot 2023-08-25T02:21:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:52 smithi121 ceph-mon[100099]: osdmap e32: 5 total, 5 up, 5 in 2023-08-25T02:21:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:52 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:53.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:52 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:21:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:52 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:21:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:52 smithi089 ceph-mon[98732]: osd.4 [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] boot 2023-08-25T02:21:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:52 smithi089 ceph-mon[98732]: osdmap e32: 5 total, 5 up, 5 in 2023-08-25T02:21:53.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:21:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- 172.21.15.122:0/3818485050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 msgr2=0x7f872c0ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3818485050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c0ffcd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8720009990 tx=0x7f8720009ca0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.137 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- 172.21.15.122:0/3818485050 shutdown_connections 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3818485050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 0x7f872c107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3818485050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c0ffcd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3818485050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- 172.21.15.122:0/3818485050 >> 172.21.15.122:0/3818485050 conn(0x7f872c0fae30 msgr2=0x7f872c0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- 172.21.15.122:0/3818485050 shutdown_connections 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- 172.21.15.122:0/3818485050 wait complete. 2023-08-25T02:21:53.138 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 Processor -- start 2023-08-25T02:21:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 -- start start 2023-08-25T02:21:53.139 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:53.145 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.135+0000 7f8732ca0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:53.145 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 0x7f872c1027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c102f30 con 0x7f872c100ab0 2023-08-25T02:21:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c1030a0 con 0x7f872c101450 2023-08-25T02:21:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c1a08c0 con 0x7f872c0ff8b0 2023-08-25T02:21:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:53.146 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33138/0 (socket says 172.21.15.122:33138) 2023-08-25T02:21:53.147 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 -- 172.21.15.122:0/3132745272 learned_addr learned my addr 172.21.15.122:0/3132745272 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:21:53.154 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f872bfff700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:53.154 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f873123d700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 0x7f872c1027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:53.154 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 -- 172.21.15.122:0/3132745272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 msgr2=0x7f872c1027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:53.154 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 0x7f872c1027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.154 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 -- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 msgr2=0x7f872c102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c102270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 -- 172.21.15.122:0/3132745272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87200095f0 con 0x7f872c0ff8b0 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f872bfff700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8730a3c700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f87200036f0 tx=0x7f8720003cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872001d070 con 0x7f872c0ff8b0 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c1a0b40 con 0x7f872c0ff8b0 2023-08-25T02:21:53.155 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f872000f460 con 0x7f872c0ff8b0 2023-08-25T02:21:53.156 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720017650 con 0x7f872c0ff8b0 2023-08-25T02:21:53.156 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.136+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c1a0fd0 con 0x7f872c0ff8b0 2023-08-25T02:21:53.166 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.137+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c107a70 con 0x7f872c0ff8b0 2023-08-25T02:21:53.167 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.138+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f872000f860 con 0x7f872c0ff8b0 2023-08-25T02:21:53.167 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.139+0000 7f8729ffb700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 0x7f871c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:21:53.167 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.139+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f87200cb4f0 con 0x7f872c0ff8b0 2023-08-25T02:21:53.167 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.139+0000 7f872bfff700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 0x7f871c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:21:53.167 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.139+0000 7f872bfff700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 0x7f871c0a2410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87180099d0 tx=0x7f8718008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:21:53.168 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.144+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8720096380 con 0x7f872c0ff8b0 2023-08-25T02:21:53.341 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:21:53.338+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f872c02ce00 con 0x7f871c09ff50 2023-08-25T02:21:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: osdmap e33: 5 total, 5 up, 5 in 2023-08-25T02:21:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: osdmap e33: 5 total, 5 up, 5 in 2023-08-25T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:54.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: osdmap e33: 5 total, 5 up, 5 in 2023-08-25T02:21:54.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:54.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:21:54.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:21:54.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:21:54.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:21:56.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:56.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/514788409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]': finished 2023-08-25T02:21:56.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: osdmap e34: 6 total, 5 up, 6 in 2023-08-25T02:21:56.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:21:56.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/514788409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]': finished 2023-08-25T02:21:56.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: osdmap e34: 6 total, 5 up, 6 in 2023-08-25T02:21:56.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:21:56.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:56.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/514788409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]: dispatch 2023-08-25T02:21:56.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9751b06-a1ca-4fe3-ba70-2dc2733e05b4"}]': finished 2023-08-25T02:21:56.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: osdmap e34: 6 total, 5 up, 6 in 2023-08-25T02:21:56.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:21:57.350 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:57 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/4210800498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:57.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:57 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/4210800498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:57.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:57 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/4210800498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:21:58.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:21:58 smithi089 ceph-mon[98732]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:21:58 smithi121 ceph-mon[100099]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:21:58.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:21:58 smithi122 ceph-mon[100457]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:00.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:00 smithi122 ceph-mon[100457]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:00.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:00 smithi089 ceph-mon[98732]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:00 smithi121 ceph-mon[100099]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:02.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:02 smithi089 ceph-mon[98732]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:02.708 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:02 smithi122 ceph-mon[100457]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:02.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:02 smithi121 ceph-mon[100099]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-08-25T02:22:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:03.637 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-08-25T02:22:03.637 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:03.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-08-25T02:22:03.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:04.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:04 smithi122 ceph-mon[100457]: Deploying daemon osd.5 on smithi122 2023-08-25T02:22:04.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:04 smithi122 ceph-mon[100457]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:04.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:04 smithi089 ceph-mon[98732]: Deploying daemon osd.5 on smithi122 2023-08-25T02:22:04.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:04 smithi089 ceph-mon[98732]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:04.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:04 smithi121 ceph-mon[100099]: Deploying daemon osd.5 on smithi122 2023-08-25T02:22:04.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:04 smithi121 ceph-mon[100099]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:06.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:06 smithi122 ceph-mon[100457]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:06.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:06 smithi089 ceph-mon[98732]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:06.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:06 smithi121 ceph-mon[100099]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:08.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:08 smithi122 ceph-mon[100457]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:08.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:08 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:08.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:08 smithi121 ceph-mon[100099]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:08 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:08.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:08 smithi089 ceph-mon[98732]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:08.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:08.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:08 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:10.355 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.350+0000 7f8729ffb700 1 -- 172.21.15.122:0/3132745272 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f872c02ce00 con 0x7f871c09ff50 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 msgr2=0x7f871c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 0x7f871c0a2410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87180099d0 tx=0x7f8718008040 comp rx=0 tx=0).stop 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 msgr2=0x7f872c103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f87200036f0 tx=0x7f8720003cb0 comp rx=0 tx=0).stop 2023-08-25T02:22:10.357 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 shutdown_connections 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f871c09ff50 0x7f871c0a2410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872c101450 0x7f872c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f872c0ff8b0 0x7f872c103c20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 --2- 172.21.15.122:0/3132745272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c100ab0 0x7f872c102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 >> 172.21.15.122:0/3132745272 conn(0x7f872c0fae30 msgr2=0x7f872c0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 shutdown_connections 2023-08-25T02:22:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:10.352+0000 7f8732ca0700 1 -- 172.21.15.122:0/3132745272 wait complete. 2023-08-25T02:22:10.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:10.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:10.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:10.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.853 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.853 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.854 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:10.855 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.5.service 2023-08-25T02:22:10.860 INFO:tasks.cephadm:Deploying osd.6 on smithi089 with /dev/vg_nvme/lv_2... 2023-08-25T02:22:10.861 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-08-25T02:22:10.901 INFO:journalctl@ceph.osd.5.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:22:10.908 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:10.909 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.909 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.909 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:10.909 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:10.910 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.910 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.914 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.914 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:22:11.631 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:22:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:22:11.325+0000 7fbb165f6640 -1 osd.5 0 log_to_monitors true 2023-08-25T02:22:11.910 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:11 smithi122 ceph-mon[100457]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:11.911 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:11 smithi122 ceph-mon[100457]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:11.911 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:11 smithi089 ceph-mon[98732]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:11.911 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:11 smithi089 ceph-mon[98732]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:11 smithi121 ceph-mon[100099]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:11 smithi121 ceph-mon[100099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-08-25T02:22:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-08-25T02:22:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-08-25T02:22:12.793 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-08-25T02:22:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-08-25T02:22:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109988 s, 953 MB/s 2023-08-25T02:22:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-08-25T02:22:12.960 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:12.961 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-08-25T02:22:12.961 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: osdmap e35: 6 total, 5 up, 6 in 2023-08-25T02:22:12.961 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:12.961 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:12.961 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-mon[100457]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:12.962 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:22:12.640+0000 7fbb0b46c700 -1 osd.5 0 waiting for initial osdmap 2023-08-25T02:22:12.962 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:22:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:22:12.646+0000 7fbb05a5e700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:22:12.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:12.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-08-25T02:22:12.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: osdmap e35: 6 total, 5 up, 6 in 2023-08-25T02:22:12.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:12.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:12.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:12 smithi121 ceph-mon[100099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:12.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:12.981 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-08-25T02:22:12.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: osdmap e35: 6 total, 5 up, 6 in 2023-08-25T02:22:12.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:12.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: from='osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:12.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:12 smithi089 ceph-mon[98732]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:22:13.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2023-08-25T02:22:13.885 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:22:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:22:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: osdmap e36: 6 total, 5 up, 6 in 2023-08-25T02:22:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:13.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:13.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:13 smithi121 ceph-mon[100099]: osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] boot 2023-08-25T02:22:13.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:22:13.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: osdmap e36: 6 total, 5 up, 6 in 2023-08-25T02:22:13.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:13.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:13.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.991 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.992 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:13.992 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:13.992 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:13.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:13.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:13.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:13 smithi122 ceph-mon[100457]: osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] boot 2023-08-25T02:22:14.076 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: osdmap e36: 6 total, 5 up, 6 in 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:14.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:14.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:14.078 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:13 smithi089 ceph-mon[98732]: osd.5 [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] boot 2023-08-25T02:22:14.916 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:22:14.917 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:22:14.917 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:14.917 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi122 2023-08-25T02:22:14.918 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 8584M 2023-08-25T02:22:14.918 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: osdmap e37: 6 total, 6 up, 6 in 2023-08-25T02:22:14.918 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:14.918 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:14 smithi089 ceph-mon[98732]: osdmap e38: 6 total, 6 up, 6 in 2023-08-25T02:22:14.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:22:14.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:22:14.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:14.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi122 2023-08-25T02:22:14.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 8584M 2023-08-25T02:22:14.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: osdmap e37: 6 total, 6 up, 6 in 2023-08-25T02:22:14.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:14.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:14 smithi121 ceph-mon[100099]: osdmap e38: 6 total, 6 up, 6 in 2023-08-25T02:22:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:22:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:22:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-08-25T02:22:14.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi122 2023-08-25T02:22:14.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 8584M 2023-08-25T02:22:14.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: osdmap e37: 6 total, 6 up, 6 in 2023-08-25T02:22:14.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:22:14.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:14 smithi122 ceph-mon[100457]: osdmap e38: 6 total, 6 up, 6 in 2023-08-25T02:22:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 -- 172.21.15.89:0/3537998997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40fea50 msgr2=0x7fb2c40feeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/3537998997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40fea50 0x7fb2c40feeb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009990 tx=0x7fb2b4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 -- 172.21.15.89:0/3537998997 shutdown_connections 2023-08-25T02:22:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/3537998997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c4108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/3537998997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 0x7fb2c41023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/3537998997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40fea50 0x7fb2c40feeb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 -- 172.21.15.89:0/3537998997 >> 172.21.15.89:0/3537998997 conn(0x7fb2c40fa5d0 msgr2=0x7fb2c40fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 -- 172.21.15.89:0/3537998997 shutdown_connections 2023-08-25T02:22:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.057+0000 7fb2c9e33700 1 -- 172.21.15.89:0/3537998997 wait complete. 2023-08-25T02:22:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 Processor -- start 2023-08-25T02:22:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 -- start start 2023-08-25T02:22:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40fea50 0x7fb2c41016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 0x7fb2c4100260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c41009e0 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c4100b50 con 0x7fb2c40fea50 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c9e33700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c41a0600 con 0x7fb2c4101fd0 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45988/0 (socket says 172.21.15.89:45988) 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 -- 172.21.15.89:0/2148372424 learned_addr learned my addr 172.21.15.89:0/2148372424 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:22:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c37fe700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40fea50 0x7fb2c41016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c3fff700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 0x7fb2c4100260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 -- 172.21.15.89:0/2148372424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 msgr2=0x7fb2c4100260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 0x7fb2c4100260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 -- 172.21.15.89:0/2148372424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40fea50 msgr2=0x7fb2c41016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40fea50 0x7fb2c41016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.058+0000 7fb2c2ffd700 1 -- 172.21.15.89:0/2148372424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40095f0 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c2ffd700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40059d0 tx=0x7fb2b40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b401d070 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c41a0880 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2b400fc40 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4017650 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.059+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c41a0ca0 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.060+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c41080e0 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.061+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fb2b4022470 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.061+0000 7fb2c0ff9700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 0x7fb2b00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.061+0000 7fb2c37fe700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 0x7fb2b00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.061+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fb2b40cc0b0 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.062+0000 7fb2c37fe700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 0x7fb2b00a2410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40680c0 tx=0x7fb2ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.065+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fb2b4096d70 con 0x7fb2c40ff3f0 2023-08-25T02:22:15.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:15.262+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2c402cd80 con 0x7fb2b009ff50 2023-08-25T02:22:16.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:16.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:16.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:16.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:16.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:16.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:16 smithi089 ceph-mon[98732]: osdmap e39: 6 total, 6 up, 6 in 2023-08-25T02:22:16.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:16 smithi121 ceph-mon[100099]: osdmap e39: 6 total, 6 up, 6 in 2023-08-25T02:22:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:16.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:16.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:16 smithi122 ceph-mon[100457]: osdmap e39: 6 total, 6 up, 6 in 2023-08-25T02:22:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:17 smithi121 ceph-mon[100099]: osdmap e40: 6 total, 6 up, 6 in 2023-08-25T02:22:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:17 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]: dispatch 2023-08-25T02:22:17.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:17 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]': finished 2023-08-25T02:22:17.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:17 smithi121 ceph-mon[100099]: osdmap e41: 7 total, 6 up, 7 in 2023-08-25T02:22:17.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:17 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:17.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:17 smithi122 ceph-mon[100457]: osdmap e40: 6 total, 6 up, 6 in 2023-08-25T02:22:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:17 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]: dispatch 2023-08-25T02:22:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:17 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]': finished 2023-08-25T02:22:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:17 smithi122 ceph-mon[100457]: osdmap e41: 7 total, 6 up, 7 in 2023-08-25T02:22:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:17 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:18.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:17 smithi089 ceph-mon[98732]: osdmap e40: 6 total, 6 up, 6 in 2023-08-25T02:22:18.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:17 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]: dispatch 2023-08-25T02:22:18.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:17 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3715664755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a77199fb-6a0a-4a5f-83a1-690b403a5791"}]': finished 2023-08-25T02:22:18.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:17 smithi089 ceph-mon[98732]: osdmap e41: 7 total, 6 up, 7 in 2023-08-25T02:22:18.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:17 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:18.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:18 smithi121 ceph-mon[100099]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:18.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:18 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/612546820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:18.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:18 smithi122 ceph-mon[100457]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:18.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:18 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/612546820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:19.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:18 smithi089 ceph-mon[98732]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:19.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:18 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/612546820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:19 smithi121 ceph-mon[100099]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2023-08-25T02:22:19.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:19 smithi122 ceph-mon[100457]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2023-08-25T02:22:20.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:19 smithi089 ceph-mon[98732]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2023-08-25T02:22:22.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:22 smithi089 ceph-mon[98732]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:22:22.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:22 smithi121 ceph-mon[100099]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:22:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:22 smithi122 ceph-mon[100457]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:22:24.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:24 smithi089 ceph-mon[98732]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-08-25T02:22:24.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-08-25T02:22:24.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:24.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:24 smithi121 ceph-mon[100099]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-08-25T02:22:24.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-08-25T02:22:24.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:24.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:24 smithi122 ceph-mon[100457]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-08-25T02:22:24.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-08-25T02:22:24.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:25.567 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:25 smithi089 ceph-mon[98732]: Deploying daemon osd.6 on smithi089 2023-08-25T02:22:25.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:25 smithi121 ceph-mon[100099]: Deploying daemon osd.6 on smithi089 2023-08-25T02:22:25.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:25 smithi122 ceph-mon[100457]: Deploying daemon osd.6 on smithi089 2023-08-25T02:22:26.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:26 smithi089 ceph-mon[98732]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-08-25T02:22:26.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:26 smithi121 ceph-mon[100099]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-08-25T02:22:26.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:26 smithi122 ceph-mon[100457]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-08-25T02:22:28.569 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:28 smithi089 ceph-mon[98732]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:22:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:28 smithi121 ceph-mon[100099]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:22:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:28 smithi122 ceph-mon[100457]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:22:29.798 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:29.990 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.991 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:29.991 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:30.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:30 smithi089 ceph-mon[98732]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:30.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:30 smithi121 ceph-mon[100099]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:30.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:30 smithi122 ceph-mon[100457]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:31.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.313+0000 7fb2c0ff9700 1 -- 172.21.15.89:0/2148372424 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb2c402cd80 con 0x7fb2b009ff50 2023-08-25T02:22:31.324 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 6 on host 'smithi089' 2023-08-25T02:22:31.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 msgr2=0x7fb2b00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:31.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 0x7fb2b00a2410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40680c0 tx=0x7fb2ac009260 comp rx=0 tx=0).stop 2023-08-25T02:22:31.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 msgr2=0x7fb2c40ffd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:31.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40059d0 tx=0x7fb2b40047a0 comp rx=0 tx=0).stop 2023-08-25T02:22:31.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 shutdown_connections 2023-08-25T02:22:31.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb2b009ff50 0x7fb2b00a2410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:31.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2c40fea50 0x7fb2c41016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:31.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb2c4101fd0 0x7fb2c4100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:31.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 --2- 172.21.15.89:0/2148372424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2c40ff3f0 0x7fb2c40ffd20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:31.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 >> 172.21.15.89:0/2148372424 conn(0x7fb2c40fa5d0 msgr2=0x7fb2c4105220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:31.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 shutdown_connections 2023-08-25T02:22:31.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:22:31.315+0000 7fb2c9e33700 1 -- 172.21.15.89:0/2148372424 wait complete. 2023-08-25T02:22:31.760 DEBUG:teuthology.orchestra.run.smithi089:osd.6> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.6.service 2023-08-25T02:22:31.798 INFO:tasks.cephadm:Deploying osd.7 on smithi121 with /dev/vg_nvme/lv_2... 2023-08-25T02:22:31.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-08-25T02:22:31.815 INFO:journalctl@ceph.osd.6.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:22:32.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:31 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:31 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:31 smithi121 ceph-mon[100099]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:32.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:31 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:31 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:31 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:31 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:31 smithi122 ceph-mon[100457]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:32.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:31 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:31 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.260 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:31 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:31 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:31 smithi089 ceph-mon[98732]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:32.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:31 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.263 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:31 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:32.382 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:22:32.541 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:22:32 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:22:32.303+0000 7f8543209640 -1 osd.6 0 log_to_monitors true 2023-08-25T02:22:33.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:32 smithi121 ceph-mon[100099]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-08-25T02:22:33.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:32 smithi122 ceph-mon[100457]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-08-25T02:22:33.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:32 smithi089 ceph-mon[98732]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-08-25T02:22:33.998 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-08-25T02:22:33.998 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-08-25T02:22:33.999 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2023-08-25T02:22:33.999 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2023-08-25T02:22:33.999 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111351 s, 942 MB/s 2023-08-25T02:22:34.000 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2023-08-25T02:22:34.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:22:33.987+0000 7f853807f700 -1 osd.6 0 waiting for initial osdmap 2023-08-25T02:22:34.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:22:34.000+0000 7f8532e72700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:22:34.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-08-25T02:22:34.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: osdmap e42: 7 total, 6 up, 7 in 2023-08-25T02:22:34.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:34.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:22:34.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:34.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:34.335 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-08-25T02:22:34.346 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: osdmap e42: 7 total, 6 up, 7 in 2023-08-25T02:22:34.346 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:34.346 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:22:34.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:34.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.356 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.356 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:34.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:34.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2023-08-25T02:22:34.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-08-25T02:22:34.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: osdmap e42: 7 total, 6 up, 7 in 2023-08-25T02:22:34.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:22:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.489 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.489 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.489 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:34.489 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:34.490 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:34.490 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:34.912 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:22:35.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi089 2023-08-25T02:22:35.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 5722M 2023-08-25T02:22:35.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:22:35.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: osdmap e43: 7 total, 6 up, 7 in 2023-08-25T02:22:35.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi089 2023-08-25T02:22:35.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 5722M 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: osdmap e43: 7 total, 6 up, 7 in 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:35.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:35.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:35.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi089 2023-08-25T02:22:35.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 5722M 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: osdmap e43: 7 total, 6 up, 7 in 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:35.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:35.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:35.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:35.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:35.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:36.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:22:36.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:22:36.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] boot 2023-08-25T02:22:36.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: osdmap e44: 7 total, 7 up, 7 in 2023-08-25T02:22:36.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:36.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:35 smithi121 ceph-mon[100099]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:36.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 -- 172.21.15.121:0/3960693104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8106730 msgr2=0x7fafb8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:36.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 --2- 172.21.15.121:0/3960693104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8106730 0x7fafb8108b40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fafb000b0a0 tx=0x7fafb000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 -- 172.21.15.121:0/3960693104 shutdown_connections 2023-08-25T02:22:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 --2- 172.21.15.121:0/3960693104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8106730 0x7fafb8108b40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 --2- 172.21.15.121:0/3960693104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 --2- 172.21.15.121:0/3960693104 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8103e00 0x7fafb81061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 -- 172.21.15.121:0/3960693104 >> 172.21.15.121:0/3960693104 conn(0x7fafb80faea0 msgr2=0x7fafb80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 -- 172.21.15.121:0/3960693104 shutdown_connections 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.245+0000 7fafbea6a700 1 -- 172.21.15.121:0/3960693104 wait complete. 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 Processor -- start 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 -- start start 2023-08-25T02:22:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb8196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:36.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:36.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8106730 0x7fafb819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:36.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819e2f0 con 0x7fafb81014d0 2023-08-25T02:22:36.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819e430 con 0x7fafb8103e00 2023-08-25T02:22:36.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbea6a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819e5c0 con 0x7fafb8106730 2023-08-25T02:22:36.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:36.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41992/0 (socket says 172.21.15.121:41992) 2023-08-25T02:22:36.267 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 -- 172.21.15.121:0/1399961138 learned_addr learned my addr 172.21.15.121:0/1399961138 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-08-25T02:22:36.267 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbc806700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb8196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:36.267 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbd007700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8106730 0x7fafb819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:36.267 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 -- 172.21.15.121:0/1399961138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8106730 msgr2=0x7fafb819dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8106730 0x7fafb819dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 -- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 msgr2=0x7fafb8196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:36.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb8196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:36.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 -- 172.21.15.121:0/1399961138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0009d00 con 0x7fafb8103e00 2023-08-25T02:22:36.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafbc806700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb8196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:22:36.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.246+0000 7fafb7fff700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c920 tx=0x7fafa400cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:36.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.247+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4007b00 con 0x7fafb8103e00 2023-08-25T02:22:36.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.247+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafa4004cf0 con 0x7fafb8103e00 2023-08-25T02:22:36.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.247+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb819e8a0 con 0x7fafb8103e00 2023-08-25T02:22:36.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.247+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40055e0 con 0x7fafb8103e00 2023-08-25T02:22:36.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.247+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb819eda0 con 0x7fafb8103e00 2023-08-25T02:22:36.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.248+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb80fca20 con 0x7fafb8103e00 2023-08-25T02:22:36.273 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.249+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fafa4007c60 con 0x7fafb8103e00 2023-08-25T02:22:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.249+0000 7fafb5ffb700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 0x7fafa80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.249+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fafa40ca160 con 0x7fafb8103e00 2023-08-25T02:22:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.249+0000 7fafbc806700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 0x7fafa80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.250+0000 7fafbc806700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 0x7fafa80a2410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fafac009cb0 tx=0x7fafac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.253+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fafa4094c00 con 0x7fafb8103e00 2023-08-25T02:22:36.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:22:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:22:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] boot 2023-08-25T02:22:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: osdmap e44: 7 total, 7 up, 7 in 2023-08-25T02:22:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:35 smithi089 ceph-mon[98732]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:36.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:36.448+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb802ce00 con 0x7fafa809ff50 2023-08-25T02:22:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:22:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:22:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: osd.6 [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] boot 2023-08-25T02:22:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: osdmap e44: 7 total, 7 up, 7 in 2023-08-25T02:22:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:22:36.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:35 smithi122 ceph-mon[100457]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-08-25T02:22:37.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:37 smithi121 ceph-mon[100099]: osdmap e45: 7 total, 7 up, 7 in 2023-08-25T02:22:37.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:37 smithi121 ceph-mon[100099]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:37.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:37 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:37.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:37 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:37.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:37 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:37.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:37 smithi089 ceph-mon[98732]: osdmap e45: 7 total, 7 up, 7 in 2023-08-25T02:22:37.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:37 smithi089 ceph-mon[98732]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:37.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:37 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:37.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:37 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:37.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:37 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:37.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:37 smithi122 ceph-mon[100457]: osdmap e45: 7 total, 7 up, 7 in 2023-08-25T02:22:37.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:37 smithi122 ceph-mon[100457]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:37.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:37 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:37.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:37 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:37.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:37 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:38.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:38 smithi089 ceph-mon[98732]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:38.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:38 smithi121 ceph-mon[100099]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:38.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:38 smithi122 ceph-mon[100457]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:39.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:39 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/3686008796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:39 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:39 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]': finished 2023-08-25T02:22:39.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:39 smithi089 ceph-mon[98732]: osdmap e46: 8 total, 7 up, 8 in 2023-08-25T02:22:39.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:39.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:39 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/3686008796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:39 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:39 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]': finished 2023-08-25T02:22:39.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:39 smithi121 ceph-mon[100099]: osdmap e46: 8 total, 7 up, 8 in 2023-08-25T02:22:39.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:39.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:39 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/3686008796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:39 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]: dispatch 2023-08-25T02:22:39.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:39 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5f512d9-eebe-40fa-beb8-f1e78c6d0af6"}]': finished 2023-08-25T02:22:39.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:39 smithi122 ceph-mon[100457]: osdmap e46: 8 total, 7 up, 8 in 2023-08-25T02:22:39.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:40.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:40 smithi089 ceph-mon[98732]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:40.333 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:40 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/1568693524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:40.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:40 smithi121 ceph-mon[100099]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:40.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:40 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/1568693524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:40.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:40 smithi122 ceph-mon[100457]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:40.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:40 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/1568693524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:22:42.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:42 smithi089 ceph-mon[98732]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:42.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:42 smithi121 ceph-mon[100099]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:42.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:42 smithi122 ceph-mon[100457]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:44.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:44 smithi089 ceph-mon[98732]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:44.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:44 smithi121 ceph-mon[100099]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:44.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:44 smithi122 ceph-mon[100457]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:46.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:46 smithi121 ceph-mon[100099]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:46.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-08-25T02:22:46.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:46.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:46 smithi121 ceph-mon[100099]: Deploying daemon osd.7 on smithi121 2023-08-25T02:22:46.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:46 smithi089 ceph-mon[98732]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:46.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-08-25T02:22:46.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:46.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:46 smithi089 ceph-mon[98732]: Deploying daemon osd.7 on smithi121 2023-08-25T02:22:46.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:46 smithi122 ceph-mon[100457]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:46.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-08-25T02:22:46.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:46.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:46 smithi122 ceph-mon[100457]: Deploying daemon osd.7 on smithi121 2023-08-25T02:22:48.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:48 smithi089 ceph-mon[98732]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:48.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:48 smithi121 ceph-mon[100099]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:48.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:48 smithi122 ceph-mon[100457]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:50 smithi121 ceph-mon[100099]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:50.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:50 smithi089 ceph-mon[98732]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:50.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:50.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:50 smithi122 ceph-mon[100457]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:52.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.407+0000 7fafb5ffb700 1 -- 172.21.15.121:0/1399961138 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fafb802ce00 con 0x7fafa809ff50 2023-08-25T02:22:52.411 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 7 on host 'smithi121' 2023-08-25T02:22:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 msgr2=0x7fafa80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 0x7fafa80a2410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fafac009cb0 tx=0x7fafac009260 comp rx=0 tx=0).stop 2023-08-25T02:22:52.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 msgr2=0x7fafb8196ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:52.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c920 tx=0x7fafa400cce0 comp rx=0 tx=0).stop 2023-08-25T02:22:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 shutdown_connections 2023-08-25T02:22:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb8106730 0x7fafb819dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafb81014d0 0x7fafb8196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fafa809ff50 0x7fafa80a2410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 --2- 172.21.15.121:0/1399961138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafb8103e00 0x7fafb8196ca0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 >> 172.21.15.121:0/1399961138 conn(0x7fafb80faea0 msgr2=0x7fafb8102140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 shutdown_connections 2023-08-25T02:22:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:22:52.410+0000 7fafbea6a700 1 -- 172.21.15.121:0/1399961138 wait complete. 2023-08-25T02:22:52.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:52 smithi089 ceph-mon[98732]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:52 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:52 smithi121 ceph-mon[100099]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:52 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:52 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:52 smithi122 ceph-mon[100457]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:52.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:52 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:52.797 DEBUG:teuthology.orchestra.run.smithi121:osd.7> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.7.service 2023-08-25T02:22:52.801 INFO:tasks.cephadm:Deploying osd.8 on smithi122 with /dev/vg_nvme/lv_2... 2023-08-25T02:22:52.801 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-08-25T02:22:52.830 INFO:journalctl@ceph.osd.7.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:22:53.369 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:22:53.632 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.632 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:53.900 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:22:53 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:22:53.691+0000 7f39a4e52640 -1 osd.7 0 log_to_monitors true 2023-08-25T02:22:54.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:54 smithi121 ceph-mon[100099]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:54.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:54 smithi121 ceph-mon[100099]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:54 smithi121 ceph-mon[100099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:54 smithi122 ceph-mon[100457]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:54.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:54 smithi122 ceph-mon[100457]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:54 smithi122 ceph-mon[100457]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:54 smithi089 ceph-mon[98732]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:54.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:54 smithi089 ceph-mon[98732]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:54 smithi089 ceph-mon[98732]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-08-25T02:22:54.977 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-08-25T02:22:54.977 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-08-25T02:22:54.977 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-08-25T02:22:54.977 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-08-25T02:22:54.978 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0142218 s, 737 MB/s 2023-08-25T02:22:54.978 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-08-25T02:22:55.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2023-08-25T02:22:55.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:22:55.404+0000 7f3999cc8700 -1 osd.7 0 waiting for initial osdmap 2023-08-25T02:22:55.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:22:55.414+0000 7f39952bc700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:22:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-08-25T02:22:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: osdmap e47: 8 total, 7 up, 8 in 2023-08-25T02:22:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:55.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-08-25T02:22:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: osdmap e47: 8 total, 7 up, 8 in 2023-08-25T02:22:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:55.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: osdmap e47: 8 total, 7 up, 8 in 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.834 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:55.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:22:55.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:55.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:22:55.835 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:22:56.072 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:22:56.655 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi121 2023-08-25T02:22:56.655 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 5722M 2023-08-25T02:22:56.655 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:56.655 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:22:56.656 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: osdmap e48: 8 total, 7 up, 8 in 2023-08-25T02:22:56.656 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:56.656 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:56 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:56.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi121 2023-08-25T02:22:56.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 5722M 2023-08-25T02:22:56.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:56.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:22:56.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: osdmap e48: 8 total, 7 up, 8 in 2023-08-25T02:22:56.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:56.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:56 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:56.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi121 2023-08-25T02:22:56.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 5722M 2023-08-25T02:22:56.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:56.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:22:56.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: osdmap e48: 8 total, 7 up, 8 in 2023-08-25T02:22:56.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:56.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:56 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.416 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 -- 172.21.15.122:0/2054830495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a8101450 msgr2=0x7f55a8107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:57.421 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 --2- 172.21.15.122:0/2054830495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a8101450 0x7f55a8107d00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f55a000b0a0 tx=0x7f55a000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.421 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 -- 172.21.15.122:0/2054830495 shutdown_connections 2023-08-25T02:22:57.422 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 --2- 172.21.15.122:0/2054830495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a80ff8b0 0x7f55a80ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.422 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 --2- 172.21.15.122:0/2054830495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a8101450 0x7f55a8107d00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.422 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 --2- 172.21.15.122:0/2054830495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8100ab0 0x7f55a8100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.423 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.408+0000 7f55ae160700 1 -- 172.21.15.122:0/2054830495 >> 172.21.15.122:0/2054830495 conn(0x7f55a80fae30 msgr2=0x7f55a80fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:22:57.424 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 -- 172.21.15.122:0/2054830495 shutdown_connections 2023-08-25T02:22:57.424 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 -- 172.21.15.122:0/2054830495 wait complete. 2023-08-25T02:22:57.424 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 Processor -- start 2023-08-25T02:22:57.425 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 -- start start 2023-08-25T02:22:57.425 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:57.425 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.409+0000 7f55ae160700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a8100ab0 0x7f55a819b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:57.435 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55ae160700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 0x7f55a819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:57.435 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55ae160700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a819e260 con 0x7f55a8101450 2023-08-25T02:22:57.435 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55ae160700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a819e3d0 con 0x7f55a8100ab0 2023-08-25T02:22:57.435 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55ae160700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a819e590 con 0x7f55a80ff8b0 2023-08-25T02:22:57.436 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:57.436 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33256/0 (socket says 172.21.15.122:33256) 2023-08-25T02:22:57.436 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 -- 172.21.15.122:0/4217311390 learned_addr learned my addr 172.21.15.122:0/4217311390 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:22:57.436 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a6ffd700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a8100ab0 0x7f55a819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:57.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a7fff700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 0x7f55a819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:57.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 -- 172.21.15.122:0/4217311390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a8100ab0 msgr2=0x7f55a819b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:57.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a8100ab0 0x7f55a819b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 -- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 msgr2=0x7f55a819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:22:57.437 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 0x7f55a819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:22:57.438 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 -- 172.21.15.122:0/4217311390 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a0009d00 con 0x7f55a80ff8b0 2023-08-25T02:22:57.438 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a7fff700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 0x7f55a819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:22:57.438 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.410+0000 7f55a77fe700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f559800d980 tx=0x7f559800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:57.438 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.411+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598004020 con 0x7f55a80ff8b0 2023-08-25T02:22:57.439 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.411+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598009730 con 0x7f55a80ff8b0 2023-08-25T02:22:57.439 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.411+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a819e870 con 0x7f55a80ff8b0 2023-08-25T02:22:57.439 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.411+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598003a80 con 0x7f55a80ff8b0 2023-08-25T02:22:57.440 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.411+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a819edc0 con 0x7f55a80ff8b0 2023-08-25T02:22:57.449 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.412+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8106e30 con 0x7f55a80ff8b0 2023-08-25T02:22:57.450 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.413+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f55980098a0 con 0x7f55a80ff8b0 2023-08-25T02:22:57.450 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.413+0000 7f55a4ff9700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 0x7f55900a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:22:57.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.413+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5430+0+0 (secure 0 0 0) 0x7f55980ca370 con 0x7f55a80ff8b0 2023-08-25T02:22:57.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.413+0000 7f55a6ffd700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 0x7f55900a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:22:57.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.414+0000 7f55a6ffd700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 0x7f55900a2490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55a80694c0 tx=0x7f559c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:22:57.453 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.421+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5598094c40 con 0x7f55a80ff8b0 2023-08-25T02:22:57.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:22:57.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:22:57.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] boot 2023-08-25T02:22:57.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: osdmap e49: 8 total, 8 up, 8 in 2023-08-25T02:22:57.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:57 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.618 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:22:57.617+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f55a802ce80 con 0x7f559009ffd0 2023-08-25T02:22:57.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:22:57.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:22:57.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] boot 2023-08-25T02:22:57.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: osdmap e49: 8 total, 8 up, 8 in 2023-08-25T02:22:57.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:57 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:22:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:22:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: osd.7 [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] boot 2023-08-25T02:22:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: osdmap e49: 8 total, 8 up, 8 in 2023-08-25T02:22:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:57 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:22:58.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:58.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: osdmap e50: 8 total, 8 up, 8 in 2023-08-25T02:22:58.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:58.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:58.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:58.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:58.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:58.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: osdmap e50: 8 total, 8 up, 8 in 2023-08-25T02:22:58.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:58.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:58.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:58.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:58.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-08-25T02:22:58.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: osdmap e50: 8 total, 8 up, 8 in 2023-08-25T02:22:58.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:22:58.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:22:58.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:22:58.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:22:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:22:59 smithi121 ceph-mon[100099]: osdmap e51: 8 total, 8 up, 8 in 2023-08-25T02:22:59.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:22:59 smithi122 ceph-mon[100457]: osdmap e51: 8 total, 8 up, 8 in 2023-08-25T02:22:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:22:59 smithi089 ceph-mon[98732]: osdmap e51: 8 total, 8 up, 8 in 2023-08-25T02:23:00.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: osdmap e52: 8 total, 8 up, 8 in 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/3150442505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]': finished 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: osdmap e53: 9 total, 8 up, 9 in 2023-08-25T02:23:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: osdmap e52: 8 total, 8 up, 8 in 2023-08-25T02:23:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/3150442505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]': finished 2023-08-25T02:23:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: osdmap e53: 9 total, 8 up, 9 in 2023-08-25T02:23:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:00.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: osdmap e52: 8 total, 8 up, 8 in 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/3150442505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]: dispatch 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3417d93a-10fe-4919-9d05-41ef7869e207"}]': finished 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: osdmap e53: 9 total, 8 up, 9 in 2023-08-25T02:23:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:01.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:01 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/3908186058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:01 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/3908186058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:01.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:01 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/3908186058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:02.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:02 smithi121 ceph-mon[100099]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:02.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:02 smithi122 ceph-mon[100457]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:02.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:02 smithi089 ceph-mon[98732]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:03.579 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:23:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:23:03.250+0000 7fa17850c700 -1 calc_pg_upmaps abort due to max <= 0 2023-08-25T02:23:04.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:04 smithi121 ceph-mon[100099]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:04 smithi122 ceph-mon[100457]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:04 smithi089 ceph-mon[98732]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:06.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:06 smithi089 ceph-mon[98732]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:06.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:06 smithi121 ceph-mon[100099]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:06.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:06 smithi122 ceph-mon[100457]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:07.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-08-25T02:23:07.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:07 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:07.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-08-25T02:23:07.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:07 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:07.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-08-25T02:23:07.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:07 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:08.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:08 smithi089 ceph-mon[98732]: Deploying daemon osd.8 on smithi122 2023-08-25T02:23:08.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:08 smithi089 ceph-mon[98732]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:08.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:08 smithi121 ceph-mon[100099]: Deploying daemon osd.8 on smithi122 2023-08-25T02:23:08.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:08 smithi121 ceph-mon[100099]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:08.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:08 smithi122 ceph-mon[100457]: Deploying daemon osd.8 on smithi122 2023-08-25T02:23:08.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:08 smithi122 ceph-mon[100457]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:10.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:10 smithi122 ceph-mon[100457]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:10.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:10 smithi089 ceph-mon[98732]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:10.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:10 smithi121 ceph-mon[100099]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:12.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:12 smithi122 ceph-mon[100457]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:12.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:12 smithi089 ceph-mon[98732]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:12.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:12.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:12 smithi121 ceph-mon[100099]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:12.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:12.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:14.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:14.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:14.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:14.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:14 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.680 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.679+0000 7f55a4ff9700 1 -- 172.21.15.122:0/4217311390 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f55a802ce80 con 0x7f559009ffd0 2023-08-25T02:23:14.681 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 8 on host 'smithi122' 2023-08-25T02:23:14.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 msgr2=0x7f55900a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:14.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 0x7f55900a2490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55a80694c0 tx=0x7f559c009260 comp rx=0 tx=0).stop 2023-08-25T02:23:14.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 msgr2=0x7f55a8068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:14.684 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f559800d980 tx=0x7f559800dc90 comp rx=0 tx=0).stop 2023-08-25T02:23:14.684 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 shutdown_connections 2023-08-25T02:23:14.684 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a8101450 0x7f55a819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:14.685 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55a80ff8b0 0x7f55a8068f60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:14.685 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f559009ffd0 0x7f55900a2490 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:14.685 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 --2- 172.21.15.122:0/4217311390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55a8100ab0 0x7f55a819b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:14.685 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 >> 172.21.15.122:0/4217311390 conn(0x7f55a80fae30 msgr2=0x7f55a8103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:23:14.685 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 shutdown_connections 2023-08-25T02:23:14.686 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:23:14.681+0000 7f55ae160700 1 -- 172.21.15.122:0/4217311390 wait complete. 2023-08-25T02:23:14.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:14.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:14.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:14 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:14 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.162 DEBUG:teuthology.orchestra.run.smithi122:osd.8> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.8.service 2023-08-25T02:23:15.165 INFO:tasks.cephadm:Deploying osd.9 on smithi089 with /dev/vg_nvme/lv_1... 2023-08-25T02:23:15.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-08-25T02:23:15.196 INFO:journalctl@ceph.osd.8.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:23:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:23:15.883 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:23:15.611+0000 7feb45075640 -1 osd.8 0 log_to_monitors true 2023-08-25T02:23:15.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:15.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-mon[100457]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:15.885 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:15 smithi122 ceph-mon[100457]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:15.899 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:15 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.899 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:15 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:15.900 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:15 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.900 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:15 smithi089 ceph-mon[98732]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:15.900 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:15 smithi089 ceph-mon[98732]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:15 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:15 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:15 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:15 smithi121 ceph-mon[100099]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:15 smithi121 ceph-mon[100099]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-08-25T02:23:16.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:16.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-08-25T02:23:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: osdmap e54: 9 total, 8 up, 9 in 2023-08-25T02:23:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:16 smithi121 ceph-mon[100099]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: osdmap e54: 9 total, 8 up, 9 in 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:16 smithi122 ceph-mon[100457]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:17.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:17.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-08-25T02:23:17.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: osdmap e54: 9 total, 8 up, 9 in 2023-08-25T02:23:17.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:17.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: from='osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:17.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:16 smithi089 ceph-mon[98732]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:23:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-08-25T02:23:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-08-25T02:23:17.149 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-08-25T02:23:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-08-25T02:23:17.149 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111907 s, 937 MB/s 2023-08-25T02:23:17.150 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-08-25T02:23:17.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:23:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:23:17.075+0000 7feb39eeb700 -1 osd.8 0 waiting for initial osdmap 2023-08-25T02:23:17.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:23:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:23:17.087+0000 7feb364e1700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:23:17.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi089:vg_nvme/lv_1 2023-08-25T02:23:18.142 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:23:18.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:23:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: osdmap e55: 9 total, 8 up, 9 in 2023-08-25T02:23:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi122 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:18.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:18.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:18 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:23:18.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: osdmap e55: 9 total, 8 up, 9 in 2023-08-25T02:23:18.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi122 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:18.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:18.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:18 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:23:18.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: osdmap e55: 9 total, 8 up, 9 in 2023-08-25T02:23:18.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:18.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-08-25T02:23:18.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi122 2023-08-25T02:23:18.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:18.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:18.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:18.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:18.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:18 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- 172.21.15.89:0/2753803911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 msgr2=0x7efd80100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 --2- 172.21.15.89:0/2753803911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd80100f10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0099c0 tx=0x7efd7c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- 172.21.15.89:0/2753803911 shutdown_connections 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 --2- 172.21.15.89:0/2753803911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 0x7efd800ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 --2- 172.21.15.89:0/2753803911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd80107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 --2- 172.21.15.89:0/2753803911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd80100f10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- 172.21.15.89:0/2753803911 >> 172.21.15.89:0/2753803911 conn(0x7efd800fae30 msgr2=0x7efd800fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- 172.21.15.89:0/2753803911 shutdown_connections 2023-08-25T02:23:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- 172.21.15.89:0/2753803911 wait complete. 2023-08-25T02:23:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 Processor -- start 2023-08-25T02:23:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 -- start start 2023-08-25T02:23:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.325+0000 7efd87a4e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 0x7efd80068fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd8019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8019e1a0 con 0x7efd80100ab0 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8019e310 con 0x7efd800ff8b0 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8019e4d0 con 0x7efd80101450 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46124/0 (socket says 172.21.15.89:46124) 2023-08-25T02:23:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 -- 172.21.15.89:0/3673538676 learned_addr learned my addr 172.21.15.89:0/3673538676 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd857ea700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 0x7efd80068fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd85feb700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd8019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 -- 172.21.15.89:0/3673538676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 msgr2=0x7efd8019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd8019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 -- 172.21.15.89:0/3673538676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 msgr2=0x7efd80068fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 0x7efd80068fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 -- 172.21.15.89:0/3673538676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0095f0 con 0x7efd80100ab0 2023-08-25T02:23:19.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd85feb700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd8019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd84fe9700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009990 tx=0x7efd7c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c01d070 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd7c004640 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8019e750 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c00f460 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.326+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8019ebf0 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.327+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd80106ea0 con 0x7efd80100ab0 2023-08-25T02:23:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.329+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7efd7c022470 con 0x7efd80100ab0 2023-08-25T02:23:19.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.329+0000 7efd767fc700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 0x7efd6c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:19.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.330+0000 7efd857ea700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 0x7efd6c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.330+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5858+0+0 (secure 0 0 0) 0x7efd7c0cc600 con 0x7efd80100ab0 2023-08-25T02:23:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.330+0000 7efd857ea700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 0x7efd6c0a2410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd70005fd0 tx=0x7efd70005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:23:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.333+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7efd7c096d10 con 0x7efd80100ab0 2023-08-25T02:23:19.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:23:19.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:23:19.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 5722M 2023-08-25T02:23:19.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] boot 2023-08-25T02:23:19.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: osdmap e56: 9 total, 9 up, 9 in 2023-08-25T02:23:19.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:19 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:19.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:23:19.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:23:19.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 5722M 2023-08-25T02:23:19.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] boot 2023-08-25T02:23:19.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: osdmap e56: 9 total, 9 up, 9 in 2023-08-25T02:23:19.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:19 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:19.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:19.531+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efd8002ce10 con 0x7efd6c09ff50 2023-08-25T02:23:19.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:23:19.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:23:19.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 5722M 2023-08-25T02:23:19.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: osd.8 [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] boot 2023-08-25T02:23:19.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: osdmap e56: 9 total, 9 up, 9 in 2023-08-25T02:23:19.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:19 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:23:20.346 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: osdmap e57: 9 total, 9 up, 9 in 2023-08-25T02:23:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:20.347 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:20.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: osdmap e57: 9 total, 9 up, 9 in 2023-08-25T02:23:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:20.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:20.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: osdmap e57: 9 total, 9 up, 9 in 2023-08-25T02:23:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:21.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:21 smithi121 ceph-mon[100099]: osdmap e58: 9 total, 9 up, 9 in 2023-08-25T02:23:21.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:21 smithi122 ceph-mon[100457]: osdmap e58: 9 total, 9 up, 9 in 2023-08-25T02:23:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:21 smithi089 ceph-mon[98732]: osdmap e58: 9 total, 9 up, 9 in 2023-08-25T02:23:22.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: osdmap e59: 9 total, 9 up, 9 in 2023-08-25T02:23:22.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:22.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]: dispatch 2023-08-25T02:23:22.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]': finished 2023-08-25T02:23:22.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: osdmap e60: 10 total, 9 up, 10 in 2023-08-25T02:23:22.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:22.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: osdmap e59: 9 total, 9 up, 9 in 2023-08-25T02:23:22.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:22.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]: dispatch 2023-08-25T02:23:22.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]': finished 2023-08-25T02:23:22.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: osdmap e60: 10 total, 9 up, 10 in 2023-08-25T02:23:22.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: osdmap e59: 9 total, 9 up, 9 in 2023-08-25T02:23:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: pgmap v178: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]: dispatch 2023-08-25T02:23:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/351273705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4d98e8-6bcc-4ca6-9fec-495733ba95d9"}]': finished 2023-08-25T02:23:22.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: osdmap e60: 10 total, 9 up, 10 in 2023-08-25T02:23:22.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:23.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:23 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/561746010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:23.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:23 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/561746010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:23.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:23 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/561746010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:24 smithi121 ceph-mon[100099]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:23:24.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:24 smithi122 ceph-mon[100457]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:23:24.584 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:24 smithi089 ceph-mon[98732]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-08-25T02:23:26.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:26 smithi089 ceph-mon[98732]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:23:26.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:26 smithi121 ceph-mon[100099]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:23:26.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:26 smithi122 ceph-mon[100457]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-08-25T02:23:28.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:28 smithi089 ceph-mon[98732]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:23:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:28 smithi121 ceph-mon[100099]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:23:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:28 smithi122 ceph-mon[100457]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-08-25T02:23:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-08-25T02:23:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:29 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:29.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-08-25T02:23:29.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:29 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:29.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-08-25T02:23:29.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:29 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:30.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:30 smithi089 ceph-mon[98732]: Deploying daemon osd.9 on smithi089 2023-08-25T02:23:30.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:30 smithi089 ceph-mon[98732]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-08-25T02:23:30.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:30 smithi121 ceph-mon[100099]: Deploying daemon osd.9 on smithi089 2023-08-25T02:23:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:30 smithi121 ceph-mon[100099]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-08-25T02:23:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:30 smithi122 ceph-mon[100457]: Deploying daemon osd.9 on smithi089 2023-08-25T02:23:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:30 smithi122 ceph-mon[100457]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-08-25T02:23:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:32 smithi089 ceph-mon[98732]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:23:32.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:32 smithi121 ceph-mon[100099]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:23:32.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:32 smithi122 ceph-mon[100457]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-08-25T02:23:34.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:34.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:34 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:34 smithi089 ceph-mon[98732]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-08-25T02:23:34.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:34 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:34 smithi121 ceph-mon[100099]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-08-25T02:23:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:34.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:34 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:34.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:34 smithi122 ceph-mon[100457]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-08-25T02:23:36.042 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 9 on host 'smithi089' 2023-08-25T02:23:36.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.041+0000 7efd767fc700 1 -- 172.21.15.89:0/3673538676 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efd8002ce10 con 0x7efd6c09ff50 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 msgr2=0x7efd6c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 0x7efd6c0a2410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd70005fd0 tx=0x7efd70005dd0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 msgr2=0x7efd8019b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009990 tx=0x7efd7c003ba0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 shutdown_connections 2023-08-25T02:23:36.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efd6c09ff50 0x7efd6c0a2410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd800ff8b0 0x7efd80068fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd80101450 0x7efd8019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 --2- 172.21.15.89:0/3673538676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd80100ab0 0x7efd8019b370 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 >> 172.21.15.89:0/3673538676 conn(0x7efd800fae30 msgr2=0x7efd80103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 shutdown_connections 2023-08-25T02:23:36.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:23:36.043+0000 7efd87a4e700 1 -- 172.21.15.89:0/3673538676 wait complete. 2023-08-25T02:23:36.413 DEBUG:teuthology.orchestra.run.smithi089:osd.9> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.9.service 2023-08-25T02:23:36.418 INFO:tasks.cephadm:Deploying osd.10 on smithi121 with /dev/vg_nvme/lv_1... 2023-08-25T02:23:36.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-08-25T02:23:36.448 INFO:journalctl@ceph.osd.9.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:23:36.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:36 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:36 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:36.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:36 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:36.987 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:23:37.206 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:23:37 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:23:37.004+0000 7f79bf6a7640 -1 osd.9 0 log_to_monitors true 2023-08-25T02:23:37.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:37 smithi121 ceph-mon[100099]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-08-25T02:23:37.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:37 smithi089 ceph-mon[98732]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-08-25T02:23:37.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:37 smithi122 ceph-mon[100457]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-08-25T02:23:38.499 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-08-25T02:23:38.499 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-08-25T02:23:38.500 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2023-08-25T02:23:38.500 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2023-08-25T02:23:38.500 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110082 s, 953 MB/s 2023-08-25T02:23:38.501 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2023-08-25T02:23:38.573 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:23:38.393+0000 7f79b451d700 -1 osd.9 0 waiting for initial osdmap 2023-08-25T02:23:38.574 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:23:38.405+0000 7f79aeb0f700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-mon[98732]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-mon[98732]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-08-25T02:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-mon[98732]: osdmap e61: 10 total, 9 up, 10 in 2023-08-25T02:23:38.574 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:38.575 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:38 smithi089 ceph-mon[98732]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:23:38.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:38 smithi121 ceph-mon[100099]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:38.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:38 smithi121 ceph-mon[100099]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-08-25T02:23:38.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:38 smithi121 ceph-mon[100099]: osdmap e61: 10 total, 9 up, 10 in 2023-08-25T02:23:38.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:38.666 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:38 smithi121 ceph-mon[100099]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:23:38.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:38 smithi122 ceph-mon[100457]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:38.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:38 smithi122 ceph-mon[100457]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-08-25T02:23:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:38 smithi122 ceph-mon[100457]: osdmap e61: 10 total, 9 up, 10 in 2023-08-25T02:23:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:38 smithi122 ceph-mon[100457]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-08-25T02:23:38.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2023-08-25T02:23:39.486 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:23:39.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:23:39.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: osdmap e62: 10 total, 9 up, 10 in 2023-08-25T02:23:39.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi089 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 4292M 2023-08-25T02:23:39.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:39.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:39.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:39 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:23:39.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: osdmap e62: 10 total, 9 up, 10 in 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi089 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 4292M 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:39.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:39 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-08-25T02:23:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: osdmap e62: 10 total, 9 up, 10 in 2023-08-25T02:23:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi089 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:23:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 4292M 2023-08-25T02:23:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:23:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:39 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:40.660 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:23:40.660 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:23:40.660 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:40.661 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] boot 2023-08-25T02:23:40.661 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: osdmap e63: 10 total, 10 up, 10 in 2023-08-25T02:23:40.661 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:23:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:23:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] boot 2023-08-25T02:23:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: osdmap e63: 10 total, 10 up, 10 in 2023-08-25T02:23:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:40.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/2914002844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc100e60 msgr2=0x7fd3dc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/2914002844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc100e60 0x7fd3dc101280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009990 tx=0x7fd3d8009ca0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/2914002844 shutdown_connections 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/2914002844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc100e60 0x7fd3dc101280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/2914002844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 0x7fd3dc107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/2914002844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/2914002844 >> 172.21.15.121:0/2914002844 conn(0x7fd3dc0fa5d0 msgr2=0x7fd3dc0fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:23:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/2914002844 shutdown_connections 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/2914002844 wait complete. 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 Processor -- start 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 -- start start 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 0x7fd3dc19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.822+0000 7fd3e3b0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc100e60 0x7fd3dc19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e3b0a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc19e0e0 con 0x7fd3dc100e60 2023-08-25T02:23:40.826 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e3b0a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc19e250 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e3b0a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc19e410 con 0x7fd3dc100480 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42130/0 (socket says 172.21.15.121:42130) 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 -- 172.21.15.121:0/3186400417 learned_addr learned my addr 172.21.15.121:0/3186400417 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e10a5700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 0x7fd3dc19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e20a7700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc100e60 0x7fd3dc19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:40.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 -- 172.21.15.121:0/3186400417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 msgr2=0x7fd3dc19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 0x7fd3dc19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 -- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc100e60 msgr2=0x7fd3dc19d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc100e60 0x7fd3dc19d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 -- 172.21.15.121:0/3186400417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80095f0 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e18a6700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000c00 tx=0x7fd3d800f780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:23:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d801d070 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3d800fb80 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc19e690 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8017650 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.823+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc19eb80 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:23:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:23:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-08-25T02:23:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: osd.9 [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] boot 2023-08-25T02:23:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: osdmap e63: 10 total, 10 up, 10 in 2023-08-25T02:23:40.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:23:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.824+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc10b090 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.825+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd3d8022470 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.826+0000 7fd3d2ffd700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 0x7fd3c80a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:23:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.826+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fd3d80cc5c0 con 0x7fd3dc0ffac0 2023-08-25T02:23:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.826+0000 7fd3e10a5700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 0x7fd3c80a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:23:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.826+0000 7fd3e10a5700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 0x7fd3c80a2490 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0ffa90 tx=0x7fd3cc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:23:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:40.829+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd3d8096be0 con 0x7fd3dc0ffac0 2023-08-25T02:23:41.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:41.024+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3dc060d60 con 0x7fd3c809ffd0 2023-08-25T02:23:41.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:41 smithi121 ceph-mon[100099]: osdmap e64: 10 total, 10 up, 10 in 2023-08-25T02:23:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:41 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:41.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:41 smithi122 ceph-mon[100457]: osdmap e64: 10 total, 10 up, 10 in 2023-08-25T02:23:41.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:41.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:41.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:41 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:42.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:41 smithi089 ceph-mon[98732]: osdmap e64: 10 total, 10 up, 10 in 2023-08-25T02:23:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:23:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:23:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:41 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:42.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:42 smithi121 ceph-mon[100099]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:42.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:42 smithi121 ceph-mon[100099]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:42.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:42 smithi122 ceph-mon[100457]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:42.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:42 smithi122 ceph-mon[100457]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:43.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:42 smithi089 ceph-mon[98732]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:23:43.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:42 smithi089 ceph-mon[98732]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:43 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/2829344573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:43.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:43 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:43.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:43 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]': finished 2023-08-25T02:23:43.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:43 smithi121 ceph-mon[100099]: osdmap e65: 11 total, 10 up, 11 in 2023-08-25T02:23:43.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:43 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:23:43.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:43 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/2829344573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:43.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:43 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:43.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:43 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]': finished 2023-08-25T02:23:43.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:43 smithi122 ceph-mon[100457]: osdmap e65: 11 total, 10 up, 11 in 2023-08-25T02:23:43.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:43 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:23:44.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:43 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/2829344573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:44.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:43 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]: dispatch 2023-08-25T02:23:44.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:43 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4dd502-8b3b-4c22-b3c3-79d177e32289"}]': finished 2023-08-25T02:23:44.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:43 smithi089 ceph-mon[98732]: osdmap e65: 11 total, 10 up, 11 in 2023-08-25T02:23:44.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:43 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:23:44.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:44 smithi121 ceph-mon[100099]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:44 smithi121 ceph-mon[100099]: from='client.? 172.21.15.121:0/4040382784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:44 smithi122 ceph-mon[100457]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:44 smithi122 ceph-mon[100457]: from='client.? 172.21.15.121:0/4040382784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:45.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:44 smithi089 ceph-mon[98732]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:45.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:44 smithi089 ceph-mon[98732]: from='client.? 172.21.15.121:0/4040382784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:23:46.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:46 smithi089 ceph-mon[98732]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:46.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:46 smithi121 ceph-mon[100099]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:46.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:46 smithi122 ceph-mon[100457]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:48.667 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:48 smithi121 ceph-mon[100099]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:48.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:48 smithi122 ceph-mon[100457]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:48.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:48 smithi089 ceph-mon[98732]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:50.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:50 smithi121 ceph-mon[100099]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:50.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-08-25T02:23:50.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:50 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:50.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:50 smithi122 ceph-mon[100457]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-08-25T02:23:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:50 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:50.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:50 smithi089 ceph-mon[98732]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:50.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-08-25T02:23:50.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:50 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:23:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:51 smithi121 ceph-mon[100099]: Deploying daemon osd.10 on smithi121 2023-08-25T02:23:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:51 smithi122 ceph-mon[100457]: Deploying daemon osd.10 on smithi121 2023-08-25T02:23:51.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:51 smithi089 ceph-mon[98732]: Deploying daemon osd.10 on smithi121 2023-08-25T02:23:52.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:52 smithi121 ceph-mon[100099]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:52 smithi122 ceph-mon[100457]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:52.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:52 smithi089 ceph-mon[98732]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:54.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:54 smithi121 ceph-mon[100099]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:54 smithi122 ceph-mon[100457]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:54.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:54 smithi089 ceph-mon[98732]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:55.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:55.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:23:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:56.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:56 smithi089 ceph-mon[98732]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:56.937 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:56 smithi121 ceph-mon[100099]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:56.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:56 smithi122 ceph-mon[100457]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:57.387 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 10 on host 'smithi121' 2023-08-25T02:23:57.387 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.385+0000 7fd3d2ffd700 1 -- 172.21.15.121:0/3186400417 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd3dc060d60 con 0x7fd3c809ffd0 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 msgr2=0x7fd3c80a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 0x7fd3c80a2490 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0ffa90 tx=0x7fd3cc00a380 comp rx=0 tx=0).stop 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 msgr2=0x7fd3dc0ff530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000c00 tx=0x7fd3d800f780 comp rx=0 tx=0).stop 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 shutdown_connections 2023-08-25T02:23:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd3c809ffd0 0x7fd3c80a2490 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3dc0ffac0 0x7fd3dc0ff530 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc100e60 0x7fd3dc19d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 --2- 172.21.15.121:0/3186400417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3dc100480 0x7fd3dc19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 >> 172.21.15.121:0/3186400417 conn(0x7fd3dc0fa5d0 msgr2=0x7fd3dc0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 shutdown_connections 2023-08-25T02:23:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-08-25T02:23:57.388+0000 7fd3e3b0a700 1 -- 172.21.15.121:0/3186400417 wait complete. 2023-08-25T02:23:57.875 DEBUG:teuthology.orchestra.run.smithi121:osd.10> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.10.service 2023-08-25T02:23:57.878 INFO:tasks.cephadm:Deploying osd.11 on smithi122 with /dev/vg_nvme/lv_1... 2023-08-25T02:23:57.878 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-08-25T02:23:57.906 INFO:journalctl@ceph.osd.10.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:23:58.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:58 smithi122 ceph-mon[100457]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:58.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.446 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:23:58.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-mon[100099]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:58 smithi089 ceph-mon[98732]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:23:58.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:23:58.939 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:23:58 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:23:58.481+0000 7f7564270640 -1 osd.10 0 log_to_monitors true 2023-08-25T02:23:59.193 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:59 smithi121 ceph-mon[100099]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:23:59.193 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:23:59 smithi121 ceph-mon[100099]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:23:59.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:59 smithi122 ceph-mon[100457]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:23:59.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:23:59 smithi122 ceph-mon[100457]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:23:59.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:59 smithi089 ceph-mon[98732]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:23:59.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:23:59 smithi089 ceph-mon[98732]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-08-25T02:24:00.009 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-08-25T02:24:00.010 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-08-25T02:24:00.010 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-08-25T02:24:00.010 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-08-25T02:24:00.010 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107499 s, 975 MB/s 2023-08-25T02:24:00.010 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-08-25T02:24:00.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-08-25T02:24:00.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: osdmap e66: 11 total, 10 up, 11 in 2023-08-25T02:24:00.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:00.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:00 smithi122 ceph-mon[100457]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:24:00.468 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:24:00.119+0000 7f75590e6700 -1 osd.10 0 waiting for initial osdmap 2023-08-25T02:24:00.468 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:24:00.132+0000 7f75536d8700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: osdmap e66: 11 total, 10 up, 11 in 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:00 smithi121 ceph-mon[100099]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:24:00.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2023-08-25T02:24:00.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-08-25T02:24:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: osdmap e66: 11 total, 10 up, 11 in 2023-08-25T02:24:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: from='osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2023-08-25T02:24:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:00 smithi089 ceph-mon[98732]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-08-25T02:24:01.044 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:24:01.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:24:01.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: osdmap e67: 11 total, 10 up, 11 in 2023-08-25T02:24:01.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi121 2023-08-25T02:24:01.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 4292M 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:01.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:01 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: osdmap e67: 11 total, 10 up, 11 in 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi121 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 4292M 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:01 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2023-08-25T02:24:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: osdmap e67: 11 total, 10 up, 11 in 2023-08-25T02:24:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi121 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 4292M 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:01.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:01.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:01.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:01 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:02.359 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 -- 172.21.15.122:0/2286072195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce481009d0 msgr2=0x7fce48100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:02.359 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 --2- 172.21.15.122:0/2286072195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce481009d0 0x7fce48100df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce38009990 tx=0x7fce38009ca0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 -- 172.21.15.122:0/2286072195 shutdown_connections 2023-08-25T02:24:02.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 --2- 172.21.15.122:0/2286072195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48102570 0x7fce48108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 --2- 172.21.15.122:0/2286072195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce481009d0 0x7fce48100df0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 --2- 172.21.15.122:0/2286072195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48101bd0 0x7fce48102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 -- 172.21.15.122:0/2286072195 >> 172.21.15.122:0/2286072195 conn(0x7fce480fbf30 msgr2=0x7fce480fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:02.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 -- 172.21.15.122:0/2286072195 shutdown_connections 2023-08-25T02:24:02.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.358+0000 7fce50074700 1 -- 172.21.15.122:0/2286072195 wait complete. 2023-08-25T02:24:02.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 Processor -- start 2023-08-25T02:24:02.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 -- start start 2023-08-25T02:24:02.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce481009d0 0x7fce481966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:02.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:02.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 0x7fce4819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:02.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4819e250 con 0x7fce481009d0 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4819e390 con 0x7fce48102570 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce50074700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4819e520 con 0x7fce48101bd0 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33398/0 (socket says 172.21.15.122:33398) 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 -- 172.21.15.122:0/327326514 learned_addr learned my addr 172.21.15.122:0/327326514 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:24:02.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4de10700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce481009d0 0x7fce481966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4e611700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 0x7fce4819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 -- 172.21.15.122:0/327326514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 msgr2=0x7fce4819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 0x7fce4819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 -- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce481009d0 msgr2=0x7fce481966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce481009d0 0x7fce481966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 -- 172.21.15.122:0/327326514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380095f0 con 0x7fce48101bd0 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4e611700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 0x7fce4819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.359+0000 7fce4d60f700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce4400d980 tx=0x7fce4400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.360+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44009b40 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.360+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce48072bb0 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.360+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce48073180 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.360+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce4400b6e0 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.360+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4400f3c0 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.361+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c005320 con 0x7fce48101bd0 2023-08-25T02:24:02.370 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.362+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fce44009ca0 con 0x7fce48101bd0 2023-08-25T02:24:02.371 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.362+0000 7fce3effd700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 0x7fce340a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:02.371 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.362+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fce440caf00 con 0x7fce48101bd0 2023-08-25T02:24:02.371 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.362+0000 7fce4de10700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 0x7fce340a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:02.371 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.363+0000 7fce4de10700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 0x7fce340a2410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce38000c00 tx=0x7fce38005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:02.372 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.365+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fce440952c0 con 0x7fce48101bd0 2023-08-25T02:24:02.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] boot 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: osdmap e68: 11 total, 11 up, 11 in 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:02.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:02 smithi121 ceph-mon[100099]: osdmap e69: 11 total, 11 up, 11 in 2023-08-25T02:24:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:24:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:24:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] boot 2023-08-25T02:24:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: osdmap e68: 11 total, 11 up, 11 in 2023-08-25T02:24:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:02.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:02.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:02 smithi122 ceph-mon[100457]: osdmap e69: 11 total, 11 up, 11 in 2023-08-25T02:24:02.564 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:02.563+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fce2c000bf0 con 0x7fce3409ff50 2023-08-25T02:24:02.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:24:02.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:24:02.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: osd.10 [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] boot 2023-08-25T02:24:02.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: osdmap e68: 11 total, 11 up, 11 in 2023-08-25T02:24:02.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:24:02.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:02.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:02 smithi089 ceph-mon[98732]: osdmap e69: 11 total, 11 up, 11 in 2023-08-25T02:24:03.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:03 smithi121 ceph-mon[100099]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:03.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:24:03.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:24:03.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:03.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:03 smithi121 ceph-mon[100099]: osdmap e70: 11 total, 11 up, 11 in 2023-08-25T02:24:03.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:03 smithi122 ceph-mon[100457]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:03.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:24:03.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:24:03.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:03.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:03 smithi122 ceph-mon[100457]: osdmap e70: 11 total, 11 up, 11 in 2023-08-25T02:24:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:03 smithi089 ceph-mon[98732]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-08-25T02:24:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-08-25T02:24:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:03 smithi089 ceph-mon[98732]: osdmap e70: 11 total, 11 up, 11 in 2023-08-25T02:24:04.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:04 smithi122 ceph-mon[100457]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:04.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:04 smithi121 ceph-mon[100099]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:04.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:04 smithi089 ceph-mon[98732]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:05.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: osdmap e71: 11 total, 11 up, 11 in 2023-08-25T02:24:05.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/2173945859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]': finished 2023-08-25T02:24:05.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: osdmap e72: 12 total, 11 up, 12 in 2023-08-25T02:24:05.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:05 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:05.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: osdmap e71: 11 total, 11 up, 11 in 2023-08-25T02:24:05.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/2173945859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]': finished 2023-08-25T02:24:05.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: osdmap e72: 12 total, 11 up, 12 in 2023-08-25T02:24:05.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:05 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: osdmap e71: 11 total, 11 up, 11 in 2023-08-25T02:24:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/2173945859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]: dispatch 2023-08-25T02:24:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e7ae39d-a06b-4a45-9bfd-b969620b53e8"}]': finished 2023-08-25T02:24:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: osdmap e72: 12 total, 11 up, 12 in 2023-08-25T02:24:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:05 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:06 smithi121 ceph-mon[100099]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:06.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:06 smithi121 ceph-mon[100099]: from='client.? 172.21.15.122:0/795399371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:24:06.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:06 smithi122 ceph-mon[100457]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:06.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:06 smithi122 ceph-mon[100457]: from='client.? 172.21.15.122:0/795399371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:24:06.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:06 smithi089 ceph-mon[98732]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:06.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:06 smithi089 ceph-mon[98732]: from='client.? 172.21.15.122:0/795399371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-08-25T02:24:08.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:08 smithi121 ceph-mon[100099]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:08.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:08 smithi122 ceph-mon[100457]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:08.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:08 smithi089 ceph-mon[98732]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:10.635 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:10 smithi122 ceph-mon[100457]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-08-25T02:24:10.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:10 smithi121 ceph-mon[100099]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-08-25T02:24:10.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:10 smithi089 ceph-mon[98732]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-08-25T02:24:12.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:12 smithi121 ceph-mon[100099]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-08-25T02:24:12.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-08-25T02:24:12.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:12 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:12.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:12 smithi122 ceph-mon[100457]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-08-25T02:24:12.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-08-25T02:24:12.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:12 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:12.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:12 smithi089 ceph-mon[98732]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-08-25T02:24:12.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-08-25T02:24:12.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:12 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:13.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:13 smithi122 ceph-mon[100457]: Deploying daemon osd.11 on smithi122 2023-08-25T02:24:13.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:13 smithi121 ceph-mon[100099]: Deploying daemon osd.11 on smithi122 2023-08-25T02:24:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:13 smithi089 ceph-mon[98732]: Deploying daemon osd.11 on smithi122 2023-08-25T02:24:14.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:14 smithi122 ceph-mon[100457]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-08-25T02:24:14.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:14 smithi121 ceph-mon[100099]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-08-25T02:24:14.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:14 smithi089 ceph-mon[98732]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-08-25T02:24:16.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:16 smithi121 ceph-mon[100099]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-08-25T02:24:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:16 smithi122 ceph-mon[100457]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-08-25T02:24:16.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:16 smithi089 ceph-mon[98732]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-08-25T02:24:18.169 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:17 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.170 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:17 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:18.170 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:17 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.170 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:17 smithi122 ceph-mon[100457]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:18.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:17 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:17 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:18.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:17 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:17 smithi121 ceph-mon[100099]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:18.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:17 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:17 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:17 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:18.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:17 smithi089 ceph-mon[98732]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:20.027 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.026+0000 7fce3effd700 1 -- 172.21.15.122:0/327326514 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fce2c000bf0 con 0x7fce3409ff50 2023-08-25T02:24:20.028 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 11 on host 'smithi122' 2023-08-25T02:24:20.029 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 msgr2=0x7fce340a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 0x7fce340a2410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce38000c00 tx=0x7fce38005fb0 comp rx=0 tx=0).stop 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 msgr2=0x7fce48196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce4400d980 tx=0x7fce4400dc90 comp rx=0 tx=0).stop 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 shutdown_connections 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce3409ff50 0x7fce340a2410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48102570 0x7fce4819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:20.030 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce481009d0 0x7fce481966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:20.031 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 --2- 172.21.15.122:0/327326514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce48101bd0 0x7fce48196c00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:20.031 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.028+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 >> 172.21.15.122:0/327326514 conn(0x7fce480fbf30 msgr2=0x7fce480fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:20.031 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.029+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 shutdown_connections 2023-08-25T02:24:20.031 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:20.029+0000 7fce50074700 1 -- 172.21.15.122:0/327326514 wait complete. 2023-08-25T02:24:20.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-mon[100457]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:20.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.543 DEBUG:teuthology.orchestra.run.smithi122:osd.11> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.11.service 2023-08-25T02:24:20.545 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-08-25T02:24:20.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd stat -f json 2023-08-25T02:24:20.573 INFO:journalctl@ceph.osd.11.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:24:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:20 smithi121 ceph-mon[100099]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:20 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:20 smithi089 ceph-mon[98732]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-08-25T02:24:20.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:20 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:21.014 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:24:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:24:20.755+0000 7f68c4c51640 -1 osd.11 0 log_to_monitors true 2023-08-25T02:24:21.112 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:24:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:21 smithi089 ceph-mon[98732]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:21 smithi089 ceph-mon[98732]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:21.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:21 smithi122 ceph-mon[100457]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:21.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:21 smithi122 ceph-mon[100457]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:21 smithi121 ceph-mon[100099]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:21 smithi121 ceph-mon[100099]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-08-25T02:24:22.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 -- 172.21.15.89:0/96803115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100230 msgr2=0x7f8010100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:22.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 --2- 172.21.15.89:0/96803115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100230 0x7f8010100690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ff80099c0 tx=0x7f7ff8009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 -- 172.21.15.89:0/96803115 shutdown_connections 2023-08-25T02:24:22.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 --2- 172.21.15.89:0/96803115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 0x7f80100ff450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 --2- 172.21.15.89:0/96803115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100bd0 0x7f8010107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 --2- 172.21.15.89:0/96803115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100230 0x7f8010100690 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 -- 172.21.15.89:0/96803115 >> 172.21.15.89:0/96803115 conn(0x7f80100fa5d0 msgr2=0x7f80100fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:22.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 -- 172.21.15.89:0/96803115 shutdown_connections 2023-08-25T02:24:22.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 -- 172.21.15.89:0/96803115 wait complete. 2023-08-25T02:24:22.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.183+0000 7f80161b3700 1 Processor -- start 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 -- start start 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 0x7f8010101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 0x7f801019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101a0370 con 0x7f8010100bd0 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101a04e0 con 0x7f80100ff030 2023-08-25T02:24:22.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f80161b3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101a06a0 con 0x7f8010100230 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46290/0 (socket says 172.21.15.89:46290) 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 -- 172.21.15.89:0/163613923 learned_addr learned my addr 172.21.15.89:0/163613923 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800effd700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 0x7f801019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800f7fe700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 0x7f8010101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 -- 172.21.15.89:0/163613923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 msgr2=0x7f801019d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:22.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 0x7f801019d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 -- 172.21.15.89:0/163613923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 msgr2=0x7f8010101fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 0x7f8010101fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 -- 172.21.15.89:0/163613923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80095f0 con 0x7f8010100bd0 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800effd700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 0x7f801019d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800ffff700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f800000e940 tx=0x7f800000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.184+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000097b0 con 0x7f8010100bd0 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.185+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80101a0980 con 0x7f8010100bd0 2023-08-25T02:24:22.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.185+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8000005020 con 0x7f8010100bd0 2023-08-25T02:24:22.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.185+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800000cbf0 con 0x7f8010100bd0 2023-08-25T02:24:22.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.185+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80101a0ea0 con 0x7f8010100bd0 2023-08-25T02:24:22.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.186+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8010106d20 con 0x7f8010100bd0 2023-08-25T02:24:22.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.187+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f8000005400 con 0x7f8010100bd0 2023-08-25T02:24:22.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.187+0000 7f800cff9700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 0x7f7ffc0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:22.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.187+0000 7f800f7fe700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 0x7f7ffc0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:22.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.187+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f8000014070 con 0x7f8010100bd0 2023-08-25T02:24:22.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.188+0000 7f800f7fe700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 0x7f7ffc0a2410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8008009550 tx=0x7f80080091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:22.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.192+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8000095190 con 0x7f8010100bd0 2023-08-25T02:24:22.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.384+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f801010a830 con 0x7f8010100bd0 2023-08-25T02:24:22.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.385+0000 7f800cff9700 1 -- 172.21.15.89:0/163613923 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7f8000005190 con 0x7f8010100bd0 2023-08-25T02:24:22.386 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:22.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 msgr2=0x7f7ffc0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 0x7f7ffc0a2410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8008009550 tx=0x7f80080091b0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 msgr2=0x7f801019fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f800000e940 tx=0x7f800000ec50 comp rx=0 tx=0).stop 2023-08-25T02:24:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 shutdown_connections 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ffc09ff50 0x7f7ffc0a2410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80100ff030 0x7f8010101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8010100bd0 0x7f801019fcc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 --2- 172.21.15.89:0/163613923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8010100230 0x7f801019d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 >> 172.21.15.89:0/163613923 conn(0x7f80100fa5d0 msgr2=0x7f80100fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 shutdown_connections 2023-08-25T02:24:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:22.387+0000 7f80161b3700 1 -- 172.21.15.89:0/163613923 wait complete. 2023-08-25T02:24:22.678 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:22.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-08-25T02:24:22.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: osdmap e73: 12 total, 11 up, 12 in 2023-08-25T02:24:22.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:22.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:22 smithi089 ceph-mon[98732]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-08-25T02:24:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: osdmap e73: 12 total, 11 up, 12 in 2023-08-25T02:24:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:22 smithi121 ceph-mon[100099]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.728 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:24:22.357+0000 7f68b9ac7700 -1 osd.11 0 waiting for initial osdmap 2023-08-25T02:24:22.728 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:24:22.372+0000 7f68b58bc700 -1 osd.11 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T02:24:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:22.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-08-25T02:24:22.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: osdmap e73: 12 total, 11 up, 12 in 2023-08-25T02:24:22.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:22.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: from='osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:22 smithi122 ceph-mon[100457]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-08-25T02:24:22.754 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1692930241,"num_in_osds":12,"osd_in_since":1692930245,"num_remapped_pgs":0} 2023-08-25T02:24:23.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: osdmap e74: 12 total, 11 up, 12 in 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/163613923' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:23 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:24:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: osdmap e74: 12 total, 11 up, 12 in 2023-08-25T02:24:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/163613923' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:23.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:23.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.734 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:23 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd stat -f json 2023-08-25T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-08-25T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: osdmap e74: 12 total, 11 up, 12 in 2023-08-25T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:23.786 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/163613923' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:23.786 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.786 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.787 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.787 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.787 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.788 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:24:23.789 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.789 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.789 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.789 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:23.791 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:23.792 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:23.792 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:23.792 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:23 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:24:24.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: purged_snaps scrub starts 2023-08-25T02:24:24.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: purged_snaps scrub ok 2023-08-25T02:24:24.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: Detected new or changed devices on smithi122 2023-08-25T02:24:24.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 4292M 2023-08-25T02:24:24.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:24.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:24.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] boot 2023-08-25T02:24:24.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: osdmap e75: 12 total, 12 up, 12 in 2023-08-25T02:24:24.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:24 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:24.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: purged_snaps scrub starts 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: purged_snaps scrub ok 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: Detected new or changed devices on smithi122 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 4292M 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] boot 2023-08-25T02:24:24.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: osdmap e75: 12 total, 12 up, 12 in 2023-08-25T02:24:24.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:24 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:24.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: purged_snaps scrub starts 2023-08-25T02:24:24.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: purged_snaps scrub ok 2023-08-25T02:24:24.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: Detected new or changed devices on smithi122 2023-08-25T02:24:24.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 4292M 2023-08-25T02:24:24.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-08-25T02:24:24.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:24.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: osd.11 [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] boot 2023-08-25T02:24:24.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: osdmap e75: 12 total, 12 up, 12 in 2023-08-25T02:24:24.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:24 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:24:25.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/1616906383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 msgr2=0x7f7b981061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:25.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/1616906383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b981061c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009990 tx=0x7f7b88009ca0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/1616906383 shutdown_connections 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/1616906383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b981014a0 0x7f7b98103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/1616906383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b98106700 0x7f7b98108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/1616906383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b981061c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/1616906383 >> 172.21.15.89:0/1616906383 conn(0x7f7b980fae30 msgr2=0x7f7b980fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:25.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/1616906383 shutdown_connections 2023-08-25T02:24:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/1616906383 wait complete. 2023-08-25T02:24:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.358+0000 7f7b9dfcf700 1 Processor -- start 2023-08-25T02:24:25.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 -- start start 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b981014a0 0x7f7b98196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b98106700 0x7f7b9819dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9819e220 con 0x7f7b98103dd0 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9819e360 con 0x7f7b98106700 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9819e4f0 con 0x7f7b981014a0 2023-08-25T02:24:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46312/0 (socket says 172.21.15.89:46312) 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 -- 172.21.15.89:0/326075995 learned_addr learned my addr 172.21.15.89:0/326075995 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b97fff700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b98106700 0x7f7b9819dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b977fe700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b981014a0 0x7f7b98196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 -- 172.21.15.89:0/326075995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b981014a0 msgr2=0x7f7b98196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b981014a0 0x7f7b98196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 -- 172.21.15.89:0/326075995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b98106700 msgr2=0x7f7b9819dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b98106700 0x7f7b9819dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 -- 172.21.15.89:0/326075995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b880095f0 con 0x7f7b98103dd0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b96ffd700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b88005910 tx=0x7f7b8800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8801d070 con 0x7f7b98103dd0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b8800fbc0 con 0x7f7b98103dd0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9819e770 con 0x7f7b98103dd0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.359+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88017650 con 0x7f7b98103dd0 2023-08-25T02:24:25.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.360+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9819ec60 con 0x7f7b98103dd0 2023-08-25T02:24:25.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.361+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b980fc9b0 con 0x7f7b98103dd0 2023-08-25T02:24:25.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.362+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f7b88022470 con 0x7f7b98103dd0 2023-08-25T02:24:25.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.362+0000 7f7b94ff9700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 0x7f7b800a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:25.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.362+0000 7f7b977fe700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 0x7f7b800a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:25.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.362+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b880cc080 con 0x7f7b98103dd0 2023-08-25T02:24:25.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.363+0000 7f7b977fe700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 0x7f7b800a2490 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c005950 tx=0x7f7b8c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:25.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.366+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7b88096340 con 0x7f7b98103dd0 2023-08-25T02:24:25.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.556+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7b9810c1d0 con 0x7f7b98103dd0 2023-08-25T02:24:25.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.557+0000 7f7b94ff9700 1 -- 172.21.15.89:0/326075995 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f7b88099960 con 0x7f7b98103dd0 2023-08-25T02:24:25.559 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 msgr2=0x7f7b800a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 0x7f7b800a2490 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c005950 tx=0x7f7b8c005cf0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 msgr2=0x7f7b98196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b88005910 tx=0x7f7b8800f710 comp rx=0 tx=0).stop 2023-08-25T02:24:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 shutdown_connections 2023-08-25T02:24:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7b8009ffd0 0x7f7b800a2490 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b98106700 0x7f7b9819dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b981014a0 0x7f7b98196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 --2- 172.21.15.89:0/326075995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b98103dd0 0x7f7b98196bd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 >> 172.21.15.89:0/326075995 conn(0x7f7b980fae30 msgr2=0x7f7b98102110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 shutdown_connections 2023-08-25T02:24:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:25.559+0000 7f7b9dfcf700 1 -- 172.21.15.89:0/326075995 wait complete. 2023-08-25T02:24:25.715 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:25 smithi089 ceph-mon[98732]: osdmap e76: 12 total, 12 up, 12 in 2023-08-25T02:24:25.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:25 smithi121 ceph-mon[100099]: osdmap e76: 12 total, 12 up, 12 in 2023-08-25T02:24:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:25 smithi122 ceph-mon[100457]: osdmap e76: 12 total, 12 up, 12 in 2023-08-25T02:24:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1692930263,"num_in_osds":12,"osd_in_since":1692930245,"num_remapped_pgs":0} 2023-08-25T02:24:25.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd dump --format=json 2023-08-25T02:24:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:24:26.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:26 smithi121 ceph-mon[100099]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:26.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:26 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/326075995' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:26.721 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:26 smithi089 ceph-mon[98732]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:26.721 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:26 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/326075995' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:26.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:26 smithi122 ceph-mon[100457]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:26.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:26 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/326075995' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-08-25T02:24:27.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 -- 172.21.15.89:0/963256048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764101300 msgr2=0x7f876410d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:27.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 --2- 172.21.15.89:0/963256048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764101300 0x7f876410d810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f875c00b0a0 tx=0x7f875c00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 -- 172.21.15.89:0/963256048 shutdown_connections 2023-08-25T02:24:27.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 --2- 172.21.15.89:0/963256048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764106940 0x7f8764106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 --2- 172.21.15.89:0/963256048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764101300 0x7f876410d810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 --2- 172.21.15.89:0/963256048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764100940 0x7f8764100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 -- 172.21.15.89:0/963256048 >> 172.21.15.89:0/963256048 conn(0x7f87640fc410 msgr2=0x7f87640fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 -- 172.21.15.89:0/963256048 shutdown_connections 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.598+0000 7f8768a55700 1 -- 172.21.15.89:0/963256048 wait complete. 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 Processor -- start 2023-08-25T02:24:27.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 -- start start 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 0x7f8764073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764106940 0x7f87640736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641a4f80 con 0x7f8764100940 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641a50f0 con 0x7f8764101300 2023-08-25T02:24:27.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f8768a55700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641a52b0 con 0x7f8764106940 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f876259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f876259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46324/0 (socket says 172.21.15.89:46324) 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.599+0000 7f876259c700 1 -- 172.21.15.89:0/3613109476 learned_addr learned my addr 172.21.15.89:0/3613109476 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f8761d9b700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 0x7f8764073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f8762d9d700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764106940 0x7f87640736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 -- 172.21.15.89:0/3613109476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764106940 msgr2=0x7f87640736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764106940 0x7f87640736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 -- 172.21.15.89:0/3613109476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 msgr2=0x7f8764073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 0x7f8764073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 -- 172.21.15.89:0/3613109476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c009d00 con 0x7f8764100940 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f8761d9b700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 0x7f8764073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f876259c700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f875800c920 tx=0x7f875800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758007b00 con 0x7f8764100940 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87641a5590 con 0x7f8764100940 2023-08-25T02:24:27.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8758004cf0 con 0x7f8764100940 2023-08-25T02:24:27.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758005520 con 0x7f8764100940 2023-08-25T02:24:27.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.600+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87641a5a90 con 0x7f8764100940 2023-08-25T02:24:27.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.601+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876404e620 con 0x7f8764100940 2023-08-25T02:24:27.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.602+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f8758004750 con 0x7f8764100940 2023-08-25T02:24:27.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.603+0000 7f87537fe700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 0x7f874c0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:27.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.603+0000 7f8761d9b700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 0x7f874c0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:27.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.603+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f87580cae10 con 0x7f8764100940 2023-08-25T02:24:27.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.603+0000 7f8761d9b700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 0x7f874c0a2490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8764074410 tx=0x7f8754005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:27.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.606+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8758094fa0 con 0x7f8764100940 2023-08-25T02:24:27.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.792+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8764074150 con 0x7f8764100940 2023-08-25T02:24:27.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.793+0000 7f87537fe700 1 -- 172.21.15.89:0/3613109476 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f87580985c0 con 0x7f8764100940 2023-08-25T02:24:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:27.795 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:24:24.361562+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:24:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 msgr2=0x7f874c0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 0x7f874c0a2490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8764074410 tx=0x7f8754005f50 comp rx=0 tx=0).stop 2023-08-25T02:24:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 msgr2=0x7f8764072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f875800c920 tx=0x7f875800cce0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 shutdown_connections 2023-08-25T02:24:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8764106940 0x7f87640736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764100940 0x7f8764072c20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f874c09ffd0 0x7f874c0a2490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 --2- 172.21.15.89:0/3613109476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8764101300 0x7f8764073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:27.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 >> 172.21.15.89:0/3613109476 conn(0x7f87640fc410 msgr2=0x7f87640fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:27.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 shutdown_connections 2023-08-25T02:24:27.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:27.795+0000 7f8768a55700 1 -- 172.21.15.89:0/3613109476 wait complete. 2023-08-25T02:24:28.194 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:24:28.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-08-25T02:24:28.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:28 smithi089 ceph-mon[98732]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:28.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:28 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3613109476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:28 smithi121 ceph-mon[100099]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:28 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3613109476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:28 smithi122 ceph-mon[100457]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:28 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3613109476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:28.750 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:24:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- 172.21.15.89:0/1359196564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 msgr2=0x7fc7280ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 --2- 172.21.15.89:0/1359196564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc7280ff5f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009990 tx=0x7fc71c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- 172.21.15.89:0/1359196564 shutdown_connections 2023-08-25T02:24:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 --2- 172.21.15.89:0/1359196564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc72810bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 --2- 172.21.15.89:0/1359196564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc7280ff5f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 --2- 172.21.15.89:0/1359196564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 0x7fc728103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- 172.21.15.89:0/1359196564 >> 172.21.15.89:0/1359196564 conn(0x7fc728075d90 msgr2=0x7fc7280fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:29.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- 172.21.15.89:0/1359196564 shutdown_connections 2023-08-25T02:24:29.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- 172.21.15.89:0/1359196564 wait complete. 2023-08-25T02:24:29.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 Processor -- start 2023-08-25T02:24:29.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.896+0000 7fc72e368700 1 -- start start 2023-08-25T02:24:29.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:29.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 0x7fc728195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:29.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc728196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:29.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281a4ec0 con 0x7fc7280ff210 2023-08-25T02:24:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281a5030 con 0x7fc7280ffbc0 2023-08-25T02:24:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72e368700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281a51f0 con 0x7fc7281042f0 2023-08-25T02:24:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46336/0 (socket says 172.21.15.89:46336) 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 -- 172.21.15.89:0/287698748 learned_addr learned my addr 172.21.15.89:0/287698748 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc7277fe700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 0x7fc728195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72c905700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc728196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 -- 172.21.15.89:0/287698748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 msgr2=0x7fc728196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc728196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 -- 172.21.15.89:0/287698748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 msgr2=0x7fc728195bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 0x7fc728195bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:29.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 -- 172.21.15.89:0/287698748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c0095f0 con 0x7fc7280ff210 2023-08-25T02:24:29.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc72c905700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc728196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:24:29.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc727fff700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc71c000c00 tx=0x7fc71c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:29.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.897+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c01d070 con 0x7fc7280ff210 2023-08-25T02:24:29.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.898+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc71c00fe80 con 0x7fc7280ff210 2023-08-25T02:24:29.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.898+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7281a5470 con 0x7fc7280ff210 2023-08-25T02:24:29.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.898+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c017650 con 0x7fc7280ff210 2023-08-25T02:24:29.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.898+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7281a58e0 con 0x7fc7280ff210 2023-08-25T02:24:29.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.898+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7281075b0 con 0x7fc7280ff210 2023-08-25T02:24:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.899+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fc71c022470 con 0x7fc7280ff210 2023-08-25T02:24:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.900+0000 7fc7257fa700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 0x7fc7100a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.900+0000 7fc7277fe700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 0x7fc7100a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.900+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc71c0cc9c0 con 0x7fc7280ff210 2023-08-25T02:24:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.900+0000 7fc7277fe700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 0x7fc7100a2490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc728102c60 tx=0x7fc718009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:29.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:29.903+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fc71c096c80 con 0x7fc7280ff210 2023-08-25T02:24:30.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.093+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc72810ef00 con 0x7fc7280ff210 2023-08-25T02:24:30.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.094+0000 7fc7257fa700 1 -- 172.21.15.89:0/287698748 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc71c09a2a0 con 0x7fc7280ff210 2023-08-25T02:24:30.095 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:24:30.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 msgr2=0x7fc7100a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:30.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 0x7fc7100a2490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc728102c60 tx=0x7fc718009260 comp rx=0 tx=0).stop 2023-08-25T02:24:30.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 msgr2=0x7fc728195680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc71c000c00 tx=0x7fc71c00f740 comp rx=0 tx=0).stop 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 shutdown_connections 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7281042f0 0x7fc728196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7280ff210 0x7fc728195680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc71009ffd0 0x7fc7100a2490 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:30.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 --2- 172.21.15.89:0/287698748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7280ffbc0 0x7fc728195bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:30.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 >> 172.21.15.89:0/287698748 conn(0x7fc728075d90 msgr2=0x7fc7280fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:30.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 shutdown_connections 2023-08-25T02:24:30.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:30.096+0000 7fc72e368700 1 -- 172.21.15.89:0/287698748 wait complete. 2023-08-25T02:24:30.492 INFO:tasks.cephadm:Adding mds.a on smithi089 2023-08-25T02:24:30.493 INFO:tasks.cephadm:Adding mds.d on smithi089 2023-08-25T02:24:30.493 INFO:tasks.cephadm:Adding mds.g on smithi089 2023-08-25T02:24:30.493 INFO:tasks.cephadm:Adding mds.j on smithi089 2023-08-25T02:24:30.494 INFO:tasks.cephadm:Adding mds.b on smithi121 2023-08-25T02:24:30.494 INFO:tasks.cephadm:Adding mds.e on smithi121 2023-08-25T02:24:30.495 INFO:tasks.cephadm:Adding mds.h on smithi121 2023-08-25T02:24:30.495 INFO:tasks.cephadm:Adding mds.k on smithi121 2023-08-25T02:24:30.495 INFO:tasks.cephadm:Adding mds.c on smithi122 2023-08-25T02:24:30.496 INFO:tasks.cephadm:Adding mds.f on smithi122 2023-08-25T02:24:30.496 INFO:tasks.cephadm:Adding mds.i on smithi122 2023-08-25T02:24:30.496 INFO:tasks.cephadm:Adding mds.l on smithi122 2023-08-25T02:24:30.497 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph orch apply mds all '12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l' 2023-08-25T02:24:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:30 smithi121 ceph-mon[100099]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:30.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:30 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/287698748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:30.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:30 smithi122 ceph-mon[100457]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:30.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:30 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/287698748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:30.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:30 smithi089 ceph-mon[98732]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:30.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:30 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/287698748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:31.074 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:24:32.372 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 -- 172.21.15.122:0/2988235950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc04101450 msgr2=0x7efc04107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:32.372 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/2988235950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc04101450 0x7efc04107d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00b0a0 tx=0x7efbfc00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.373 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 -- 172.21.15.122:0/2988235950 shutdown_connections 2023-08-25T02:24:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/2988235950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc040ff8b0 0x7efc040ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/2988235950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc04101450 0x7efc04107d00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/2988235950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04100ab0 0x7efc04100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 -- 172.21.15.122:0/2988235950 >> 172.21.15.122:0/2988235950 conn(0x7efc040fae30 msgr2=0x7efc040fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:32.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 -- 172.21.15.122:0/2988235950 shutdown_connections 2023-08-25T02:24:32.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.371+0000 7efc0a5ae700 1 -- 172.21.15.122:0/2988235950 wait complete. 2023-08-25T02:24:32.380 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 Processor -- start 2023-08-25T02:24:32.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 -- start start 2023-08-25T02:24:32.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:32.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc04100ab0 0x7efc04196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:32.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 0x7efc0419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:32.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0419e2e0 con 0x7efc04101450 2023-08-25T02:24:32.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0419e420 con 0x7efc04100ab0 2023-08-25T02:24:32.383 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc0a5ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0419e5b0 con 0x7efc040ff8b0 2023-08-25T02:24:32.383 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:32.383 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33530/0 (socket says 172.21.15.122:33530) 2023-08-25T02:24:32.383 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 -- 172.21.15.122:0/1036961148 learned_addr learned my addr 172.21.15.122:0/1036961148 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-08-25T02:24:32.384 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc037fe700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc04100ab0 0x7efc04196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:32.384 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc08b4b700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 0x7efc0419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:32.384 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 -- 172.21.15.122:0/1036961148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc04100ab0 msgr2=0x7efc04196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:32.384 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc04100ab0 0x7efc04196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 -- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 msgr2=0x7efc0419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 0x7efc0419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc03fff700 1 -- 172.21.15.122:0/1036961148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc009d00 con 0x7efc040ff8b0 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.372+0000 7efc08b4b700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 0x7efc0419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc03fff700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efbf400d9b0 tx=0x7efbf400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4004020 con 0x7efc040ff8b0 2023-08-25T02:24:32.385 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbf4009730 con 0x7efc040ff8b0 2023-08-25T02:24:32.386 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4003a30 con 0x7efc040ff8b0 2023-08-25T02:24:32.386 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0419e890 con 0x7efc040ff8b0 2023-08-25T02:24:32.386 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.373+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0419ed40 con 0x7efc040ff8b0 2023-08-25T02:24:32.386 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.374+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0410c280 con 0x7efc040ff8b0 2023-08-25T02:24:32.386 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.375+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7efbf40098a0 con 0x7efc040ff8b0 2023-08-25T02:24:32.387 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.375+0000 7efc017fa700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 0x7efbec0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:32.387 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.375+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efbf40caa40 con 0x7efc040ff8b0 2023-08-25T02:24:32.387 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.376+0000 7efc037fe700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 0x7efbec0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:32.387 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.376+0000 7efc037fe700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 0x7efbec0a2410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efbf8007be0 tx=0x7efbf80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:32.387 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.380+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7efbf40cce60 con 0x7efc040ff8b0 2023-08-25T02:24:32.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:32 smithi122 ceph-mon[100457]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:32.577 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.576+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}) v1 -- 0x7efc04060e60 con 0x7efbec09ff50 2023-08-25T02:24:32.585 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.584+0000 7efc017fa700 1 -- 172.21.15.122:0/1036961148 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7efc04060e60 con 0x7efbec09ff50 2023-08-25T02:24:32.586 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mds.all update... 2023-08-25T02:24:32.588 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 msgr2=0x7efbec0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:32.588 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 0x7efbec0a2410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efbf8007be0 tx=0x7efbf80073d0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.588 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 msgr2=0x7efc04196750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efbf400d9b0 tx=0x7efbf400dcc0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 shutdown_connections 2023-08-25T02:24:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc04101450 0x7efc0419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc040ff8b0 0x7efc04196750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.589 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7efbec09ff50 0x7efbec0a2410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.590 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 --2- 172.21.15.122:0/1036961148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc04100ab0 0x7efc04196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:32.590 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 >> 172.21.15.122:0/1036961148 conn(0x7efc040fae30 msgr2=0x7efc040fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:32.590 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 shutdown_connections 2023-08-25T02:24:32.591 INFO:teuthology.orchestra.run.smithi122.stderr:2023-08-25T02:24:32.587+0000 7efc0a5ae700 1 -- 172.21.15.122:0/1036961148 wait complete. 2023-08-25T02:24:32.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:32 smithi121 ceph-mon[100099]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:32.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:32 smithi089 ceph-mon[98732]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:33.129 DEBUG:teuthology.orchestra.run.smithi089:mds.a> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.a.service 2023-08-25T02:24:33.133 DEBUG:teuthology.orchestra.run.smithi089:mds.d> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.d.service 2023-08-25T02:24:33.170 INFO:journalctl@ceph.mds.a.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:24:33.171 DEBUG:teuthology.orchestra.run.smithi089:mds.g> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.g.service 2023-08-25T02:24:33.199 INFO:journalctl@ceph.mds.d.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:24:33.200 DEBUG:teuthology.orchestra.run.smithi089:mds.j> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.j.service 2023-08-25T02:24:33.227 INFO:journalctl@ceph.mds.g.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:24:33.228 DEBUG:teuthology.orchestra.run.smithi121:mds.b> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.b.service 2023-08-25T02:24:33.230 DEBUG:teuthology.orchestra.run.smithi121:mds.e> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.e.service 2023-08-25T02:24:33.254 INFO:journalctl@ceph.mds.j.smithi089.stdout:-- Logs begin at Fri 2023-08-25 02:00:21 UTC. -- 2023-08-25T02:24:33.267 INFO:journalctl@ceph.mds.b.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:24:33.268 DEBUG:teuthology.orchestra.run.smithi121:mds.h> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.h.service 2023-08-25T02:24:33.298 INFO:journalctl@ceph.mds.e.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:24:33.298 DEBUG:teuthology.orchestra.run.smithi121:mds.k> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.k.service 2023-08-25T02:24:33.326 INFO:journalctl@ceph.mds.h.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:24:33.327 DEBUG:teuthology.orchestra.run.smithi122:mds.c> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.c.service 2023-08-25T02:24:33.330 DEBUG:teuthology.orchestra.run.smithi122:mds.f> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.f.service 2023-08-25T02:24:33.353 INFO:journalctl@ceph.mds.k.smithi121.stdout:-- Logs begin at Fri 2023-08-25 02:01:01 UTC. -- 2023-08-25T02:24:33.359 INFO:journalctl@ceph.mds.c.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:24:33.360 DEBUG:teuthology.orchestra.run.smithi122:mds.i> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.i.service 2023-08-25T02:24:33.387 INFO:journalctl@ceph.mds.f.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:24:33.388 DEBUG:teuthology.orchestra.run.smithi122:mds.l> sudo journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.l.service 2023-08-25T02:24:33.414 INFO:journalctl@ceph.mds.i.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:24:33.415 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-08-25T02:24:33.416 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:24:33.442 INFO:journalctl@ceph.mds.l.smithi122.stdout:-- Logs begin at Fri 2023-08-25 02:01:59 UTC. -- 2023-08-25T02:24:33.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 -- 172.21.15.89:0/1354813588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c101390 msgr2=0x7ff20c107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 --2- 172.21.15.89:0/1354813588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c101390 0x7ff20c107c40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff20400b0d0 tx=0x7ff20400b3e0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 -- 172.21.15.89:0/1354813588 shutdown_connections 2023-08-25T02:24:33.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 --2- 172.21.15.89:0/1354813588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c101390 0x7ff20c107c40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 --2- 172.21.15.89:0/1354813588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c0ff800 0x7ff20c0ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 --2- 172.21.15.89:0/1354813588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c1009f0 0x7ff20c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 -- 172.21.15.89:0/1354813588 >> 172.21.15.89:0/1354813588 conn(0x7ff20c0fad50 msgr2=0x7ff20c0fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:33.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 -- 172.21.15.89:0/1354813588 shutdown_connections 2023-08-25T02:24:33.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.544+0000 7ff213a7e700 1 -- 172.21.15.89:0/1354813588 wait complete. 2023-08-25T02:24:33.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 Processor -- start 2023-08-25T02:24:33.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 -- start start 2023-08-25T02:24:33.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c0ff800 0x7ff20c068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c101390 0x7ff20c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c19e230 con 0x7ff20c1009f0 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c19e3a0 con 0x7ff20c0ff800 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff213a7e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c19e560 con 0x7ff20c101390 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46350/0 (socket says 172.21.15.89:46350) 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 -- 172.21.15.89:0/2927643980 learned_addr learned my addr 172.21.15.89:0/2927643980 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff21201b700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c101390 0x7ff20c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 -- 172.21.15.89:0/2927643980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c101390 msgr2=0x7ff20c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c101390 0x7ff20c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff21181a700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c0ff800 0x7ff20c068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 -- 172.21.15.89:0/2927643980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c0ff800 msgr2=0x7ff20c068610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c0ff800 0x7ff20c068610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.545+0000 7ff211019700 1 -- 172.21.15.89:0/2927643980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204009d00 con 0x7ff20c1009f0 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff211019700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff20000d950 tx=0x7ff20000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200009940 con 0x7ff20c1009f0 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff200010460 con 0x7ff20c1009f0 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c19e840 con 0x7ff20c1009f0 2023-08-25T02:24:33.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000f620 con 0x7ff20c1009f0 2023-08-25T02:24:33.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.546+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c19ed50 con 0x7ff20c1009f0 2023-08-25T02:24:33.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.547+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c106d50 con 0x7ff20c1009f0 2023-08-25T02:24:33.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.548+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7ff2000108d0 con 0x7ff20c1009f0 2023-08-25T02:24:33.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.548+0000 7ff1feffd700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 0x7ff1f80a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.549+0000 7ff21181a700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 0x7ff1f80a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.549+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2000cac00 con 0x7ff20c1009f0 2023-08-25T02:24:33.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.549+0000 7ff21181a700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 0x7ff1f80a2460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff20c100850 tx=0x7ff208009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:33.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.552+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff200094e30 con 0x7ff20c1009f0 2023-08-25T02:24:33.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:33.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2023-08-25T02:24:33.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:33.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:33.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:33.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:33.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:33 smithi122 ceph-mon[100457]: Deploying daemon mds.l on smithi122 2023-08-25T02:24:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.746+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff20c10b250 con 0x7ff20c1009f0 2023-08-25T02:24:33.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.747+0000 7ff1feffd700 1 -- 172.21.15.89:0/2927643980 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18410 (secure 0 0 0) 0x7ff200098450 con 0x7ff20c1009f0 2023-08-25T02:24:33.748 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:33.749 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:24:24.361562+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:24:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 msgr2=0x7ff1f80a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 0x7ff1f80a2460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff20c100850 tx=0x7ff208009260 comp rx=0 tx=0).stop 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 msgr2=0x7ff20c19b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff20000d950 tx=0x7ff20000dc60 comp rx=0 tx=0).stop 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 shutdown_connections 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff1f809ffa0 0x7ff1f80a2460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff20c0ff800 0x7ff20c068610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff20c101390 0x7ff20c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 --2- 172.21.15.89:0/2927643980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff20c1009f0 0x7ff20c19b410 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 >> 172.21.15.89:0/2927643980 conn(0x7ff20c0fad50 msgr2=0x7ff20c103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 shutdown_connections 2023-08-25T02:24:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.749+0000 7ff213a7e700 1 -- 172.21.15.89:0/2927643980 wait complete. 2023-08-25T02:24:33.761 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:33.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2023-08-25T02:24:33.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:33.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:33.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:33.764 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:33.764 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.764 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:33 smithi089 ceph-mon[98732]: Deploying daemon mds.l on smithi122 2023-08-25T02:24:33.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:24:33.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:24:33.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- 172.21.15.89:0/3364196351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c841013a0 msgr2=0x7f8c84107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 --2- 172.21.15.89:0/3364196351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c841013a0 0x7f8c84107c50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00b0d0 tx=0x7f8c7c00b3e0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- 172.21.15.89:0/3364196351 shutdown_connections 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 --2- 172.21.15.89:0/3364196351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c841013a0 0x7f8c84107c50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 --2- 172.21.15.89:0/3364196351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 0x7f8c840ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 --2- 172.21.15.89:0/3364196351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c84100a00 0x7f8c84100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- 172.21.15.89:0/3364196351 >> 172.21.15.89:0/3364196351 conn(0x7f8c840fad80 msgr2=0x7f8c840fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:33.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- 172.21.15.89:0/3364196351 shutdown_connections 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- 172.21.15.89:0/3364196351 wait complete. 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 Processor -- start 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.890+0000 7f8c8afef700 1 -- start start 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 0x7f8c841037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c841013a0 0x7f8c84102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84102ac0 con 0x7f8c84100a00 2023-08-25T02:24:33.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84102c30 con 0x7f8c841013a0 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c841a0760 con 0x7f8c840ff810 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46364/0 (socket says 172.21.15.89:46364) 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 -- 172.21.15.89:0/2986355750 learned_addr learned my addr 172.21.15.89:0/2986355750 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c88d8b700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 0x7f8c841037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8958c700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c841013a0 0x7f8c84102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 -- 172.21.15.89:0/2986355750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 msgr2=0x7f8c841037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 0x7f8c841037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 -- 172.21.15.89:0/2986355750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c841013a0 msgr2=0x7f8c84102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c841013a0 0x7f8c84102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 -- 172.21.15.89:0/2986355750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c009d00 con 0x7f8c84100a00 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c83fff700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d950 tx=0x7f8c7000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70009940 con 0x7f8c84100a00 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c70010460 con 0x7f8c84100a00 2023-08-25T02:24:33.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c841a0a40 con 0x7f8c84100a00 2023-08-25T02:24:33.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.891+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000f620 con 0x7f8c84100a00 2023-08-25T02:24:33.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.892+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c841a0f50 con 0x7f8c84100a00 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.893+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8410b030 con 0x7f8c84100a00 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.893+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f8c700108d0 con 0x7f8c84100a00 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.894+0000 7f8c81ffb700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 0x7f8c740a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.894+0000 7f8c88d8b700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 0x7f8c740a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.894+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8c700cac90 con 0x7f8c84100a00 2023-08-25T02:24:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.894+0000 7f8c88d8b700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 0x7f8c740a2490 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c84100860 tx=0x7f8c7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:33.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:33.898+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8c70094e20 con 0x7f8c84100a00 2023-08-25T02:24:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:24:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:33 smithi121 ceph-mon[100099]: Deploying daemon mds.l on smithi122 2023-08-25T02:24:34.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.087+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c841a11a0 con 0x7f8c84100a00 2023-08-25T02:24:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.088+0000 7f8c81ffb700 1 -- 172.21.15.89:0/2986355750 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c70098440 con 0x7f8c84100a00 2023-08-25T02:24:34.089 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:24:34.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.090+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 msgr2=0x7f8c740a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.090+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 0x7f8c740a2490 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8c84100860 tx=0x7f8c7800b410 comp rx=0 tx=0).stop 2023-08-25T02:24:34.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.090+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 msgr2=0x7f8c84101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.090+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000d950 tx=0x7f8c7000dc60 comp rx=0 tx=0).stop 2023-08-25T02:24:34.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 shutdown_connections 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8c7409ffd0 0x7f8c740a2490 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c841013a0 0x7f8c84102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840ff810 0x7f8c841037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 --2- 172.21.15.89:0/2986355750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100a00 0x7f8c84101de0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 >> 172.21.15.89:0/2986355750 conn(0x7f8c840fad80 msgr2=0x7f8c84067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 shutdown_connections 2023-08-25T02:24:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.091+0000 7f8c8afef700 1 -- 172.21.15.89:0/2986355750 wait complete. 2023-08-25T02:24:34.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:34.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 -- 172.21.15.89:0/3314981064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 msgr2=0x7fe680100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 --2- 172.21.15.89:0/3314981064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680100e50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0099c0 tx=0x7fe67c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 -- 172.21.15.89:0/3314981064 shutdown_connections 2023-08-25T02:24:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 --2- 172.21.15.89:0/3314981064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6800ff800 0x7fe6800ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 --2- 172.21.15.89:0/3314981064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe680101390 0x7fe680107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.224+0000 7fe68823e700 1 --2- 172.21.15.89:0/3314981064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680100e50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.225+0000 7fe68823e700 1 -- 172.21.15.89:0/3314981064 >> 172.21.15.89:0/3314981064 conn(0x7fe6800fad50 msgr2=0x7fe6800fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.225+0000 7fe68823e700 1 -- 172.21.15.89:0/3314981064 shutdown_connections 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.225+0000 7fe68823e700 1 -- 172.21.15.89:0/3314981064 wait complete. 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.225+0000 7fe68823e700 1 Processor -- start 2023-08-25T02:24:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.225+0000 7fe68823e700 1 -- start start 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6800ff800 0x7fe680195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 0x7fe68019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68019cfa0 con 0x7fe6801009f0 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68019d0e0 con 0x7fe680101390 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68019d270 con 0x7fe6800ff800 2023-08-25T02:24:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe685fda700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6800ff800 0x7fe680195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46376/0 (socket says 172.21.15.89:46376) 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 -- 172.21.15.89:0/811965383 learned_addr learned my addr 172.21.15.89:0/811965383 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6867db700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 0x7fe68019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 -- 172.21.15.89:0/811965383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6800ff800 msgr2=0x7fe680195420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6800ff800 0x7fe680195420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 -- 172.21.15.89:0/811965383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 msgr2=0x7fe68019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 0x7fe68019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 -- 172.21.15.89:0/811965383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0095f0 con 0x7fe6801009f0 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6867db700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 0x7fe68019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe6857d9700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009fd0 tx=0x7fe67c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c01d070 con 0x7fe6801009f0 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe67c0054f0 con 0x7fe6801009f0 2023-08-25T02:24:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68019d4f0 con 0x7fe6801009f0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c004bd0 con 0x7fe6801009f0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.226+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68019d910 con 0x7fe6801009f0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.227+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68010b040 con 0x7fe6801009f0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.228+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fe67c00fb80 con 0x7fe6801009f0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.229+0000 7fe676ffd700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 0x7fe66c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.229+0000 7fe685fda700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 0x7fe66c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.229+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe67c0cc080 con 0x7fe6801009f0 2023-08-25T02:24:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.229+0000 7fe685fda700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 0x7fe66c0a2460 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe680100850 tx=0x7fe67000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.232+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fe67c096340 con 0x7fe6801009f0 2023-08-25T02:24:34.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.467+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe680106680 con 0x7fe6801009f0 2023-08-25T02:24:34.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.467+0000 7fe676ffd700 1 -- 172.21.15.89:0/811965383 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fe67c099960 con 0x7fe6801009f0 2023-08-25T02:24:34.468 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:34.469 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-08-25T02:24:34.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 msgr2=0x7fe66c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 0x7fe66c0a2460 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe680100850 tx=0x7fe67000a3b0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 msgr2=0x7fe680195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009fd0 tx=0x7fe67c00f690 comp rx=0 tx=0).stop 2023-08-25T02:24:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 shutdown_connections 2023-08-25T02:24:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe66c09ffa0 0x7fe66c0a2460 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe680101390 0x7fe68019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6800ff800 0x7fe680195420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 --2- 172.21.15.89:0/811965383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6801009f0 0x7fe680195960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 >> 172.21.15.89:0/811965383 conn(0x7fe6800fad50 msgr2=0x7fe6800fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 shutdown_connections 2023-08-25T02:24:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:34.469+0000 7fe68823e700 1 -- 172.21.15.89:0/811965383 wait complete. 2023-08-25T02:24:34.473 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 1 2023-08-25T02:24:34.612 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:34 smithi122 systemd[1]: Starting Ceph mds.l for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:34 smithi089 ceph-mon[98732]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2927643980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2986355750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/811965383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:34.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:34 smithi121 ceph-mon[100099]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:34.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2927643980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2986355750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/811965383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:34.977 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:34 smithi122 podman[125563]: 2023-08-25T02:24:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-mon[100457]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2927643980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:24:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2986355750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:24:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/811965383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:35.478 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-mds[125603]: -- 172.21.15.122:0/2249360099 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a77d3d51e0 con 0x55a77d3aac00 2023-08-25T02:24:35.478 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:34 smithi122 bash[125563]: 6295cb31265c154526759439b25ea9519b40d71cb3127c4f5cb25a1c6873e0b0 2023-08-25T02:24:35.479 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-l[125579]: starting mds.l at 2023-08-25T02:24:35.479 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:24:35 smithi122 systemd[1]: Started Ceph mds.l for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:36.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6832/1630095418,v1:172.21.15.122:6833/1630095418] up:boot 2023-08-25T02:24:36.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: fsmap 1 up:standby 2023-08-25T02:24:36.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:24:36.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: Deploying daemon mds.k on smithi121 2023-08-25T02:24:36.072 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:35 smithi121 ceph-mon[100099]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:36.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6832/1630095418,v1:172.21.15.122:6833/1630095418] up:boot 2023-08-25T02:24:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: fsmap 1 up:standby 2023-08-25T02:24:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:24:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:36.331 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: Deploying daemon mds.k on smithi121 2023-08-25T02:24:36.332 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:35 smithi089 ceph-mon[98732]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6832/1630095418,v1:172.21.15.122:6833/1630095418] up:boot 2023-08-25T02:24:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: fsmap 1 up:standby 2023-08-25T02:24:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:24:36.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: Deploying daemon mds.k on smithi121 2023-08-25T02:24:36.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:35 smithi122 ceph-mon[100457]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:36.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:36.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 -- 172.21.15.89:0/884311837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 msgr2=0x7f5b1c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:36.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/884311837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c100d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5b100099c0 tx=0x7f5b10009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 -- 172.21.15.89:0/884311837 shutdown_connections 2023-08-25T02:24:36.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/884311837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/884311837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c100d60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/884311837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c101b30 0x7f5b1c101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 -- 172.21.15.89:0/884311837 >> 172.21.15.89:0/884311837 conn(0x7f5b1c0fbed0 msgr2=0x7f5b1c0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:36.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 -- 172.21.15.89:0/884311837 shutdown_connections 2023-08-25T02:24:36.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 -- 172.21.15.89:0/884311837 wait complete. 2023-08-25T02:24:36.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.617+0000 7f5b21fcf700 1 Processor -- start 2023-08-25T02:24:36.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 -- start start 2023-08-25T02:24:36.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c101b30 0x7f5b1c196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19e1a0 con 0x7f5b1c100940 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19e2e0 con 0x7f5b1c101b30 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b21fcf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19e470 con 0x7f5b1c1024d0 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:36.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46388/0 (socket says 172.21.15.89:46388) 2023-08-25T02:24:36.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 -- 172.21.15.89:0/2064190248 learned_addr learned my addr 172.21.15.89:0/2064190248 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:36.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1affd700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c101b30 0x7f5b1c196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:36.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1bfff700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:36.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 -- 172.21.15.89:0/2064190248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 msgr2=0x7f5b1c19dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:36.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c19dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 -- 172.21.15.89:0/2064190248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c101b30 msgr2=0x7f5b1c196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:36.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c101b30 0x7f5b1c196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 -- 172.21.15.89:0/2064190248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100095f0 con 0x7f5b1c100940 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1bfff700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b1b7fe700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009990 tx=0x7f5b10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.618+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1001d070 con 0x7f5b1c100940 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.619+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b10004090 con 0x7f5b1c100940 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.619+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c19e6f0 con 0x7f5b1c100940 2023-08-25T02:24:36.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.619+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10022d00 con 0x7f5b1c100940 2023-08-25T02:24:36.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.619+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c19eba0 con 0x7f5b1c100940 2023-08-25T02:24:36.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.619+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c107f00 con 0x7f5b1c100940 2023-08-25T02:24:36.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.621+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5b10022470 con 0x7f5b1c100940 2023-08-25T02:24:36.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.621+0000 7f5b18ff9700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 0x7f5b040a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:36.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.621+0000 7f5b1affd700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 0x7f5b040a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:36.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.621+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5b100cc5c0 con 0x7f5b1c100940 2023-08-25T02:24:36.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.622+0000 7f5b1affd700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 0x7f5b040a2460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c005950 tx=0x7f5b0c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:36.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.624+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5b10096880 con 0x7f5b1c100940 2023-08-25T02:24:36.718 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:36 smithi121 systemd[1]: Starting Ceph mds.k for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:36.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.858+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1c10c150 con 0x7f5b1c100940 2023-08-25T02:24:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.859+0000 7f5b18ff9700 1 -- 172.21.15.89:0/2064190248 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f5b10022e60 con 0x7f5b1c100940 2023-08-25T02:24:36.860 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:36.860 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-08-25T02:24:36.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 msgr2=0x7f5b040a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:36.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 0x7f5b040a2460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c005950 tx=0x7f5b0c00a300 comp rx=0 tx=0).stop 2023-08-25T02:24:36.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 msgr2=0x7f5b1c196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:36.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009990 tx=0x7f5b10004730 comp rx=0 tx=0).stop 2023-08-25T02:24:36.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 shutdown_connections 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b1c1024d0 0x7f5b1c19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b1c100940 0x7f5b1c196620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5b0409ffa0 0x7f5b040a2460 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 --2- 172.21.15.89:0/2064190248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b1c101b30 0x7f5b1c196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 >> 172.21.15.89:0/2064190248 conn(0x7f5b1c0fbed0 msgr2=0x7f5b1c105040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:36.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 shutdown_connections 2023-08-25T02:24:36.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:36.861+0000 7f5b21fcf700 1 -- 172.21.15.89:0/2064190248 wait complete. 2023-08-25T02:24:36.864 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 2 2023-08-25T02:24:37.202 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:37 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2064190248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:37.202 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:36 smithi121 podman[124234]: 2023-08-25T02:24:37.202 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:37 smithi121 bash[124234]: d22661d9f62e1652ff5ea17458bc5dbca1bff45466324358a32028faf6041bbc 2023-08-25T02:24:37.202 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:37 smithi121 ceph-mds[124275]: -- 172.21.15.121:0/1721672760 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a1862b1e0 con 0x556a18600c00 2023-08-25T02:24:37.203 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-k[124250]: starting mds.k at 2023-08-25T02:24:37.468 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:24:37 smithi121 systemd[1]: Started Ceph mds.k for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:37.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:37 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2064190248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:37.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:37 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2064190248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:38.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: Deploying daemon mds.j on smithi089 2023-08-25T02:24:38.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:38.384 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6832/2625413815,v1:172.21.15.121:6833/2625413815] up:boot 2023-08-25T02:24:38.384 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: fsmap 2 up:standby 2023-08-25T02:24:38.384 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:38 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:24:38.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:38.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:38.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:38.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: Deploying daemon mds.j on smithi089 2023-08-25T02:24:38.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:38.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6832/2625413815,v1:172.21.15.121:6833/2625413815] up:boot 2023-08-25T02:24:38.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: fsmap 2 up:standby 2023-08-25T02:24:38.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:38 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:24:38.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:38.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: Deploying daemon mds.j on smithi089 2023-08-25T02:24:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6832/2625413815,v1:172.21.15.121:6833/2625413815] up:boot 2023-08-25T02:24:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: fsmap 2 up:standby 2023-08-25T02:24:38.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:38 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:24:38.829 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:38 smithi089 systemd[1]: Starting Ceph mds.j for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:38.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:39.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3802307438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b28100930 msgr2=0x7f0b28100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:39.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3802307438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b28100930 0x7f0b28100d50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0b180099c0 tx=0x7f0b18009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3802307438 shutdown_connections 2023-08-25T02:24:39.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3802307438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b281024c0 0x7f0b28108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3802307438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b28100930 0x7f0b28100d50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3802307438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28101b20 0x7f0b28101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3802307438 >> 172.21.15.89:0/3802307438 conn(0x7f0b280fbe60 msgr2=0x7f0b280fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:39.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3802307438 shutdown_connections 2023-08-25T02:24:39.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.012+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3802307438 wait complete. 2023-08-25T02:24:39.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 Processor -- start 2023-08-25T02:24:39.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 -- start start 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28100930 0x7f0b280746d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b28101b20 0x7f0b28072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280739f0 con 0x7f0b281024c0 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28073b60 con 0x7f0b28101b20 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b2dea6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281a0660 con 0x7f0b28100930 2023-08-25T02:24:39.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:39.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46404/0 (socket says 172.21.15.89:46404) 2023-08-25T02:24:39.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b27fff700 1 -- 172.21.15.89:0/3189709164 learned_addr learned my addr 172.21.15.89:0/3189709164 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:39.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.013+0000 7f0b26ffd700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b28101b20 0x7f0b28072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:39.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b277fe700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28100930 0x7f0b280746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:39.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 -- 172.21.15.89:0/3189709164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28100930 msgr2=0x7f0b280746d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:39.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28100930 0x7f0b280746d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 -- 172.21.15.89:0/3189709164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b28101b20 msgr2=0x7f0b28072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:39.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b28101b20 0x7f0b28072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 -- 172.21.15.89:0/3189709164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180095f0 con 0x7f0b281024c0 2023-08-25T02:24:39.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b27fff700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007ee0 tx=0x7f0b2000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:39.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20017070 con 0x7f0b281024c0 2023-08-25T02:24:39.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2000f040 con 0x7f0b281024c0 2023-08-25T02:24:39.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b281a0940 con 0x7f0b281024c0 2023-08-25T02:24:39.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20005740 con 0x7f0b281024c0 2023-08-25T02:24:39.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.014+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b281a0e50 con 0x7f0b281024c0 2023-08-25T02:24:39.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.015+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28107ea0 con 0x7f0b281024c0 2023-08-25T02:24:39.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.016+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0b2000dc60 con 0x7f0b281024c0 2023-08-25T02:24:39.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.017+0000 7f0b24ff9700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 0x7f0b140a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:39.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.017+0000 7f0b277fe700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 0x7f0b140a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:39.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.017+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0b200ca5e0 con 0x7f0b281024c0 2023-08-25T02:24:39.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.017+0000 7f0b277fe700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 0x7f0b140a24e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800b580 tx=0x7f0b18005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:39.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.021+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0b20094770 con 0x7f0b281024c0 2023-08-25T02:24:39.191 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:38 smithi089 podman[131974]: 2023-08-25T02:24:39.192 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:39 smithi089 bash[131974]: 09d18e7bf1d3e2506766a2838e7f84fd4032fbc76e6b9bbf7c3385c747a8b5ea 2023-08-25T02:24:39.192 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:39 smithi089 ceph-mds[132057]: -- 172.21.15.89:0/1411393398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571379851e0 con 0x55713795b000 2023-08-25T02:24:39.192 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:39 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-j[132013]: starting mds.j at 2023-08-25T02:24:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.270+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b2810c1c0 con 0x7f0b281024c0 2023-08-25T02:24:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.271+0000 7f0b24ff9700 1 -- 172.21.15.89:0/3189709164 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f0b20097d90 con 0x7f0b281024c0 2023-08-25T02:24:39.273 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:39.273 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-08-25T02:24:39.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 msgr2=0x7f0b140a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:39.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 0x7f0b140a24e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800b580 tx=0x7f0b18005e80 comp rx=0 tx=0).stop 2023-08-25T02:24:39.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 msgr2=0x7f0b28073250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:39.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007ee0 tx=0x7f0b2000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 shutdown_connections 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b281024c0 0x7f0b28073250 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b28100930 0x7f0b280746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0b140a0020 0x7f0b140a24e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 --2- 172.21.15.89:0/3189709164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b28101b20 0x7f0b28072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.273+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 >> 172.21.15.89:0/3189709164 conn(0x7f0b280fbe60 msgr2=0x7f0b28104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:39.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.274+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 shutdown_connections 2023-08-25T02:24:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:39.274+0000 7f0b2dea6700 1 -- 172.21.15.89:0/3189709164 wait complete. 2023-08-25T02:24:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 4 2023-08-25T02:24:39.579 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:24:39 smithi089 systemd[1]: Started Ceph mds.j for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:40.540 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.540 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.89:6834/1349178421,v1:172.21.15.89:6835/1349178421] up:boot 2023-08-25T02:24:40.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: fsmap 3 up:standby 2023-08-25T02:24:40.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:24:40.542 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3189709164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:40.542 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:40.542 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.542 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:40.543 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:40.543 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:40.544 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:40 smithi089 ceph-mon[98732]: Deploying daemon mds.d on smithi089 2023-08-25T02:24:40.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.89:6834/1349178421,v1:172.21.15.89:6835/1349178421] up:boot 2023-08-25T02:24:40.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: fsmap 3 up:standby 2023-08-25T02:24:40.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:24:40.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3189709164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:40.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:40 smithi121 ceph-mon[100099]: Deploying daemon mds.d on smithi089 2023-08-25T02:24:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.89:6834/1349178421,v1:172.21.15.89:6835/1349178421] up:boot 2023-08-25T02:24:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: fsmap 3 up:standby 2023-08-25T02:24:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:24:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3189709164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:40.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:40.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:40.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:40.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:40.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:40 smithi122 ceph-mon[100457]: Deploying daemon mds.d on smithi089 2023-08-25T02:24:40.830 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:40 smithi089 systemd[1]: Starting Ceph mds.d for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:41.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:41.329 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:41 smithi089 podman[132357]: 2023-08-25T02:24:41.330 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:41 smithi089 bash[132357]: 3fe4c700567c3bb1a2f7ca734503bf015c004d12835439d5c642fb328e1de392 2023-08-25T02:24:41.330 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:41 smithi089 ceph-mds[132397]: -- 172.21.15.89:0/4152643624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b13c9c31e0 con 0x55b13c998c00 2023-08-25T02:24:41.330 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:41 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-d[132373]: starting mds.d at 2023-08-25T02:24:41.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 -- 172.21.15.89:0/4127700124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb14100180 msgr2=0x7fdb141005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:41.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/4127700124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb14100180 0x7fdb141005e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009990 tx=0x7fdb08009ca0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 -- 172.21.15.89:0/4127700124 shutdown_connections 2023-08-25T02:24:41.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/4127700124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb140fef90 0x7fdb140ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/4127700124 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 0x7fdb141073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/4127700124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb14100180 0x7fdb141005e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.434+0000 7fdb19ce3700 1 -- 172.21.15.89:0/4127700124 >> 172.21.15.89:0/4127700124 conn(0x7fdb140fa4e0 msgr2=0x7fdb140fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:41.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- 172.21.15.89:0/4127700124 shutdown_connections 2023-08-25T02:24:41.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- 172.21.15.89:0/4127700124 wait complete. 2023-08-25T02:24:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 Processor -- start 2023-08-25T02:24:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- start start 2023-08-25T02:24:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 0x7fdb14196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 0x7fdb1419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:41.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419e0a0 con 0x7fdb140fef90 2023-08-25T02:24:41.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419e1e0 con 0x7fdb14100180 2023-08-25T02:24:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.435+0000 7fdb19ce3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419e370 con 0x7fdb14100b20 2023-08-25T02:24:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46436/0 (socket says 172.21.15.89:46436) 2023-08-25T02:24:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 -- 172.21.15.89:0/2295463998 learned_addr learned my addr 172.21.15.89:0/2295463998 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb12ffd700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 0x7fdb14196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb13fff700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 0x7fdb1419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 -- 172.21.15.89:0/2295463998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 msgr2=0x7fdb1419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:41.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 0x7fdb1419dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 -- 172.21.15.89:0/2295463998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 msgr2=0x7fdb14196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:41.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 0x7fdb14196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 -- 172.21.15.89:0/2295463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb080095f0 con 0x7fdb140fef90 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb12ffd700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 0x7fdb14196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb137fe700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c970 tx=0x7fdb0400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04004220 con 0x7fdb140fef90 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb04007d60 con 0x7fdb140fef90 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04003d00 con 0x7fdb140fef90 2023-08-25T02:24:41.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1419e650 con 0x7fdb140fef90 2023-08-25T02:24:41.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.436+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1419eb10 con 0x7fdb140fef90 2023-08-25T02:24:41.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.437+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1410c1e0 con 0x7fdb140fef90 2023-08-25T02:24:41.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.438+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fdb04004380 con 0x7fdb140fef90 2023-08-25T02:24:41.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.439+0000 7fdb10ff9700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 0x7fdafc0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:41.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.439+0000 7fdb12ffd700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 0x7fdafc0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:41.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.439+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb040cadb0 con 0x7fdb140fef90 2023-08-25T02:24:41.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.439+0000 7fdb12ffd700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 0x7fdafc0a2460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb08006010 tx=0x7fdb0800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:41.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.442+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fdb04094f40 con 0x7fdb140fef90 2023-08-25T02:24:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.679+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb1419ef20 con 0x7fdb140fef90 2023-08-25T02:24:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.679+0000 7fdb10ff9700 1 -- 172.21.15.89:0/2295463998 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7fdb04098560 con 0x7fdb140fef90 2023-08-25T02:24:41.681 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:41.681 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-08-25T02:24:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 msgr2=0x7fdafc0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 0x7fdafc0a2460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb08006010 tx=0x7fdb0800b540 comp rx=0 tx=0).stop 2023-08-25T02:24:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 msgr2=0x7fdb14196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400c970 tx=0x7fdb0400cd30 comp rx=0 tx=0).stop 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 shutdown_connections 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb14100b20 0x7fdb1419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb140fef90 0x7fdb14196520 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdafc09ffa0 0x7fdafc0a2460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 --2- 172.21.15.89:0/2295463998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100180 0x7fdb14196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 >> 172.21.15.89:0/2295463998 conn(0x7fdb140fa4e0 msgr2=0x7fdb140fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 shutdown_connections 2023-08-25T02:24:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:41.681+0000 7fdb19ce3700 1 -- 172.21.15.89:0/2295463998 wait complete. 2023-08-25T02:24:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 5 2023-08-25T02:24:41.693 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:24:41 smithi089 systemd[1]: Started Ceph mds.d for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:42.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:42.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.89:6836/826072159,v1:172.21.15.89:6837/826072159] up:boot 2023-08-25T02:24:42.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: fsmap 4 up:standby 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:42.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:42.705 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:42.705 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: Deploying daemon mds.e on smithi121 2023-08-25T02:24:42.705 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:42 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2295463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:42.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:42.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.89:6836/826072159,v1:172.21.15.89:6837/826072159] up:boot 2023-08-25T02:24:42.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: fsmap 4 up:standby 2023-08-25T02:24:42.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:42.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: Deploying daemon mds.e on smithi121 2023-08-25T02:24:42.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:42 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2295463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:42.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.89:6836/826072159,v1:172.21.15.89:6837/826072159] up:boot 2023-08-25T02:24:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: fsmap 4 up:standby 2023-08-25T02:24:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:24:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: Deploying daemon mds.e on smithi121 2023-08-25T02:24:42.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:42 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2295463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:43.333 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:42 smithi121 systemd[1]: Starting Ceph mds.e for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:43.621 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:43 smithi121 podman[124575]: 2023-08-25T02:24:43.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 -- 172.21.15.89:0/848216091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64081001c0 msgr2=0x7f6408100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 --2- 172.21.15.89:0/848216091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64081001c0 0x7f6408100620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f63f800aad0 tx=0x7f63f800ade0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 -- 172.21.15.89:0/848216091 shutdown_connections 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 --2- 172.21.15.89:0/848216091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6408100b60 0x7f6408107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 --2- 172.21.15.89:0/848216091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64080fefd0 0x7f64080ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 --2- 172.21.15.89:0/848216091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64081001c0 0x7f6408100620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 -- 172.21.15.89:0/848216091 >> 172.21.15.89:0/848216091 conn(0x7f64080fa520 msgr2=0x7f64080fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 -- 172.21.15.89:0/848216091 shutdown_connections 2023-08-25T02:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.834+0000 7f640d6f8700 1 -- 172.21.15.89:0/848216091 wait complete. 2023-08-25T02:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 Processor -- start 2023-08-25T02:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 -- start start 2023-08-25T02:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 0x7f6408196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6408100b60 0x7f640819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640819e130 con 0x7f64080fefd0 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640819e270 con 0x7f64081001c0 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f640d6f8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640819e400 con 0x7f6408100b60 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46450/0 (socket says 172.21.15.89:46450) 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 -- 172.21.15.89:0/2142330725 learned_addr learned my addr 172.21.15.89:0/2142330725 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f64067fc700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 0x7f6408196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f64077fe700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6408100b60 0x7f640819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 -- 172.21.15.89:0/2142330725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6408100b60 msgr2=0x7f640819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6408100b60 0x7f640819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.835+0000 7f6406ffd700 1 -- 172.21.15.89:0/2142330725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 msgr2=0x7f6408196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f6406ffd700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 0x7f6408196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f6406ffd700 1 -- 172.21.15.89:0/2142330725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f800a700 con 0x7f64080fefd0 2023-08-25T02:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f64067fc700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 0x7f6408196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f6406ffd700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00d980 tx=0x7f63fc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00de60 con 0x7f64080fefd0 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63fc0044e0 con 0x7f64080fefd0 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640819e6e0 con 0x7f64080fefd0 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003be0 con 0x7f64080fefd0 2023-08-25T02:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.836+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640819ec40 con 0x7f64080fefd0 2023-08-25T02:24:43.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.837+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64081064f0 con 0x7f64080fefd0 2023-08-25T02:24:43.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.838+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f63fc009bb0 con 0x7f64080fefd0 2023-08-25T02:24:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.838+0000 7f63effff700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 0x7f63f00a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.838+0000 7f64067fc700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 0x7f63f00a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.839+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f63fc0cae40 con 0x7f64080fefd0 2023-08-25T02:24:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.839+0000 7f64067fc700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 0x7f63f00a24e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000c00 tx=0x7f63f8005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:43.842+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f63fc0950f0 con 0x7f64080fefd0 2023-08-25T02:24:43.968 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:43 smithi121 ceph-mds[124615]: -- 172.21.15.121:0/3612173693 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f5b4a2f1e0 con 0x55f5b4a04800 2023-08-25T02:24:43.969 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:43 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: starting mds.e at 2023-08-25T02:24:43.969 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:43 smithi121 bash[124575]: c8768201ed8b0e3d390d00da362c05118c342ea1c33a51434aeee30eb12506b5 2023-08-25T02:24:43.969 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:24:43 smithi121 systemd[1]: Started Ceph mds.e for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:44.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.080+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f640802ce20 con 0x7f64080fefd0 2023-08-25T02:24:44.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.081+0000 7f63effff700 1 -- 172.21.15.89:0/2142330725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7f63fc017e90 con 0x7f64080fefd0 2023-08-25T02:24:44.082 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:44.082 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 msgr2=0x7f63f00a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 0x7f63f00a24e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000c00 tx=0x7f63f8005e80 comp rx=0 tx=0).stop 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 msgr2=0x7f64081965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00d980 tx=0x7f63fc00dd40 comp rx=0 tx=0).stop 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 shutdown_connections 2023-08-25T02:24:44.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6408100b60 0x7f640819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fefd0 0x7f64081965b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f63f00a0020 0x7f63f00a24e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 --2- 172.21.15.89:0/2142330725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64081001c0 0x7f6408196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 >> 172.21.15.89:0/2142330725 conn(0x7f64080fa520 msgr2=0x7f6408103630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 shutdown_connections 2023-08-25T02:24:44.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:44.083+0000 7f640d6f8700 1 -- 172.21.15.89:0/2142330725 wait complete. 2023-08-25T02:24:44.086 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 5 2023-08-25T02:24:44.643 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:44.643 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.644 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.644 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.644 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:44.644 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:44.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:44.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:44 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2142330725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:44.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:44.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:44 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2142330725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:44.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:44 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2142330725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:45.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:45 smithi121 ceph-mon[100099]: Deploying daemon mds.f on smithi122 2023-08-25T02:24:45.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:45 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6834/217468698,v1:172.21.15.121:6835/217468698] up:boot 2023-08-25T02:24:45.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:45 smithi121 ceph-mon[100099]: fsmap 5 up:standby 2023-08-25T02:24:45.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:45 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:24:45.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:45 smithi122 ceph-mon[100457]: Deploying daemon mds.f on smithi122 2023-08-25T02:24:45.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:45 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6834/217468698,v1:172.21.15.121:6835/217468698] up:boot 2023-08-25T02:24:45.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:45 smithi122 ceph-mon[100457]: fsmap 5 up:standby 2023-08-25T02:24:45.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:45 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:24:45.730 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:45 smithi122 systemd[1]: Starting Ceph mds.f for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:45 smithi089 ceph-mon[98732]: Deploying daemon mds.f on smithi122 2023-08-25T02:24:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:45 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6834/217468698,v1:172.21.15.121:6835/217468698] up:boot 2023-08-25T02:24:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:45 smithi089 ceph-mon[98732]: fsmap 5 up:standby 2023-08-25T02:24:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:45 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:24:46.028 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:45 smithi122 podman[125906]: 2023-08-25T02:24:46.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:46.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 -- 172.21.15.89:0/2727992047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60101b20 msgr2=0x7f1a60101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:46.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/2727992047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60101b20 0x7f1a60101f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0099c0 tx=0x7f1a5c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 -- 172.21.15.89:0/2727992047 shutdown_connections 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/2727992047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a60100930 0x7f1a60100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/2727992047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a601024c0 0x7f1a60108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/2727992047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60101b20 0x7f1a60101f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 -- 172.21.15.89:0/2727992047 >> 172.21.15.89:0/2727992047 conn(0x7f1a600fbe60 msgr2=0x7f1a600fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:46.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 -- 172.21.15.89:0/2727992047 shutdown_connections 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.229+0000 7f1a67c1f700 1 -- 172.21.15.89:0/2727992047 wait complete. 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 Processor -- start 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 -- start start 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a60101b20 0x7f1a60196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a601024c0 0x7f1a6019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6019e150 con 0x7f1a60100930 2023-08-25T02:24:46.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6019e290 con 0x7f1a601024c0 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a67c1f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6019e420 con 0x7f1a60101b20 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a659bb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a659bb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46464/0 (socket says 172.21.15.89:46464) 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a659bb700 1 -- 172.21.15.89:0/707995246 learned_addr learned my addr 172.21.15.89:0/707995246 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.230+0000 7f1a651ba700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a60101b20 0x7f1a60196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:46.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 -- 172.21.15.89:0/707995246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a60101b20 msgr2=0x7f1a60196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a60101b20 0x7f1a60196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 -- 172.21.15.89:0/707995246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a601024c0 msgr2=0x7f1a6019db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a601024c0 0x7f1a6019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 -- 172.21.15.89:0/707995246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58009c40 con 0x7f1a60100930 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a659bb700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d980 tx=0x7f1a5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50004020 con 0x7f1a60100930 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a500096b0 con 0x7f1a60100930 2023-08-25T02:24:46.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50003a50 con 0x7f1a60100930 2023-08-25T02:24:46.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0095f0 con 0x7f1a60100930 2023-08-25T02:24:46.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.231+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6019ea30 con 0x7f1a60100930 2023-08-25T02:24:46.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.232+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60107ed0 con 0x7f1a60100930 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.233+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f1a50009820 con 0x7f1a60100930 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.234+0000 7f1a56ffd700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 0x7f1a4c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.234+0000 7f1a651ba700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 0x7f1a4c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.234+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1a500cad80 con 0x7f1a60100930 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.234+0000 7f1a651ba700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 0x7f1a4c0a2460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.237+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1a50094f10 con 0x7f1a60100930 2023-08-25T02:24:46.393 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mds[125946]: -- 172.21.15.122:0/399188419 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580882131e0 con 0x5580881e9000 2023-08-25T02:24:46.393 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: starting mds.f at 2023-08-25T02:24:46.393 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:46 smithi122 bash[125906]: a289a00d90b3e2f7abe7ff5f7100e9e1f0da775df9adc26f85576b9893a9abab 2023-08-25T02:24:46.393 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:24:46 smithi122 systemd[1]: Started Ceph mds.f for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:46.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.472+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a6002cc20 con 0x7f1a60100930 2023-08-25T02:24:46.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.473+0000 7f1a56ffd700 1 -- 172.21.15.89:0/707995246 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5286 (secure 0 0 0) 0x7f1a50098530 con 0x7f1a60100930 2023-08-25T02:24:46.474 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:46.474 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2023-08-25T02:24:46.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 msgr2=0x7f1a4c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:46.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 0x7f1a4c0a2460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c019040 comp rx=0 tx=0).stop 2023-08-25T02:24:46.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 msgr2=0x7f1a601965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:46.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d980 tx=0x7f1a5000dc90 comp rx=0 tx=0).stop 2023-08-25T02:24:46.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 shutdown_connections 2023-08-25T02:24:46.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a4c09ffa0 0x7f1a4c0a2460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a601024c0 0x7f1a6019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a60100930 0x7f1a601965d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 --2- 172.21.15.89:0/707995246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a60101b20 0x7f1a60196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:46.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 >> 172.21.15.89:0/707995246 conn(0x7f1a600fbe60 msgr2=0x7f1a60105010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:46.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 shutdown_connections 2023-08-25T02:24:46.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:46.475+0000 7f1a67c1f700 1 -- 172.21.15.89:0/707995246 wait complete. 2023-08-25T02:24:46.480 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 7 2023-08-25T02:24:46.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:46.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:46.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:46 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:46.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:46.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:46.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:46 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:46.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:46.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:46 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:47.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:47 smithi122 ceph-mon[100457]: Deploying daemon mds.c on smithi122 2023-08-25T02:24:47.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:47 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6834/3100558972,v1:172.21.15.122:6835/3100558972] up:boot 2023-08-25T02:24:47.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:47 smithi122 ceph-mon[100457]: fsmap 6 up:standby 2023-08-25T02:24:47.711 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:47 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:24:47.711 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:47 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/707995246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:47.711 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:47 smithi122 systemd[1]: Starting Ceph mds.c for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:47.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:47 smithi121 ceph-mon[100099]: Deploying daemon mds.c on smithi122 2023-08-25T02:24:47.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:47 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6834/3100558972,v1:172.21.15.122:6835/3100558972] up:boot 2023-08-25T02:24:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:47 smithi121 ceph-mon[100099]: fsmap 6 up:standby 2023-08-25T02:24:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:47 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:24:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:47 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/707995246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:47 smithi089 ceph-mon[98732]: Deploying daemon mds.c on smithi122 2023-08-25T02:24:47.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:47 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6834/3100558972,v1:172.21.15.122:6835/3100558972] up:boot 2023-08-25T02:24:47.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:47 smithi089 ceph-mon[98732]: fsmap 6 up:standby 2023-08-25T02:24:47.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:47 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:24:47.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:47 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/707995246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:48.274 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:48 smithi122 podman[126246]: 2023-08-25T02:24:48.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:48.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 -- 172.21.15.89:0/1006401205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad240ff690 msgr2=0x7fad240ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:48.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 --2- 172.21.15.89:0/1006401205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad240ff690 0x7fad240ffab0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad140099c0 tx=0x7fad14009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 -- 172.21.15.89:0/1006401205 shutdown_connections 2023-08-25T02:24:48.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 --2- 172.21.15.89:0/1006401205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad24101220 0x7fad24107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 --2- 172.21.15.89:0/1006401205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad240ff690 0x7fad240ffab0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 --2- 172.21.15.89:0/1006401205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad24100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 -- 172.21.15.89:0/1006401205 >> 172.21.15.89:0/1006401205 conn(0x7fad240fabe0 msgr2=0x7fad240fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:48.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 -- 172.21.15.89:0/1006401205 shutdown_connections 2023-08-25T02:24:48.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.621+0000 7fad28d9e700 1 -- 172.21.15.89:0/1006401205 wait complete. 2023-08-25T02:24:48.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 Processor -- start 2023-08-25T02:24:48.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 -- start start 2023-08-25T02:24:48.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad240ff690 0x7fad24068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:48.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad2419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:48.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:48.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419de80 con 0x7fad24101220 2023-08-25T02:24:48.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419dff0 con 0x7fad240ff690 2023-08-25T02:24:48.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad28d9e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419e1b0 con 0x7fad24100880 2023-08-25T02:24:48.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad23fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:48.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad23fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46482/0 (socket says 172.21.15.89:46482) 2023-08-25T02:24:48.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad22ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad2419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:48.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad23fff700 1 -- 172.21.15.89:0/3065350060 learned_addr learned my addr 172.21.15.89:0/3065350060 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:48.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.622+0000 7fad237fe700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad240ff690 0x7fad24068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:48.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 -- 172.21.15.89:0/3065350060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 msgr2=0x7fad2419b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:48.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad2419b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 -- 172.21.15.89:0/3065350060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad240ff690 msgr2=0x7fad24068610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:48.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad240ff690 0x7fad24068610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 -- 172.21.15.89:0/3065350060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad140095f0 con 0x7fad24101220 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad22ffd700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad2419b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad23fff700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00ef30 tx=0x7fad1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c018070 con 0x7fad24101220 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad1c00f040 con 0x7fad24101220 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2419e430 con 0x7fad24101220 2023-08-25T02:24:48.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00cd00 con 0x7fad24101220 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.623+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2419e8f0 con 0x7fad24101220 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.624+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad241064d0 con 0x7fad24101220 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.625+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fad1c003a00 con 0x7fad24101220 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.625+0000 7fad20ff9700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 0x7fad0c0a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.626+0000 7fad237fe700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 0x7fad0c0a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.626+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fad1c0cb330 con 0x7fad24101220 2023-08-25T02:24:48.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.626+0000 7fad237fe700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 0x7fad0c0a2500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad14009990 tx=0x7fad14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:48.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.629+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fad1c095550 con 0x7fad24101220 2023-08-25T02:24:48.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:48.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:48.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:48 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:48.728 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mds[126286]: -- 172.21.15.122:0/2177516918 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589486411e0 con 0x558948617000 2023-08-25T02:24:48.728 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: starting mds.c at 2023-08-25T02:24:48.728 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:48 smithi122 bash[126246]: 85c3f14932dbbb3bb4fc1e8ae7169d065a707f12570b3489df7eab84c9162ed3 2023-08-25T02:24:48.728 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:24:48 smithi122 systemd[1]: Started Ceph mds.c for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:48.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:48.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:48 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:48.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:48.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:48.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:48.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:48.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:48 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:48.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.863+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2402cdc0 con 0x7fad24101220 2023-08-25T02:24:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.863+0000 7fad20ff9700 1 -- 172.21.15.89:0/3065350060 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7fad1c0b8070 con 0x7fad24101220 2023-08-25T02:24:48.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:48.865 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2023-08-25T02:24:48.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 msgr2=0x7fad0c0a2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 0x7fad0c0a2500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad14009990 tx=0x7fad14005fb0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 msgr2=0x7fad2419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00ef30 tx=0x7fad1c00c3a0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 shutdown_connections 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad0c0a0040 0x7fad0c0a2500 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad240ff690 0x7fad24068610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad24101220 0x7fad2419d6b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 --2- 172.21.15.89:0/3065350060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad24100880 0x7fad2419b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:48.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.865+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 >> 172.21.15.89:0/3065350060 conn(0x7fad240fabe0 msgr2=0x7fad240fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:48.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.866+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 shutdown_connections 2023-08-25T02:24:48.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:48.866+0000 7fad28d9e700 1 -- 172.21.15.89:0/3065350060 wait complete. 2023-08-25T02:24:48.868 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-08-25T02:24:49.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:49 smithi121 ceph-mon[100099]: Deploying daemon mds.h on smithi121 2023-08-25T02:24:49.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:49 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6836/2359799361,v1:172.21.15.122:6837/2359799361] up:boot 2023-08-25T02:24:49.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:49 smithi121 ceph-mon[100099]: fsmap 7 up:standby 2023-08-25T02:24:49.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:49 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:24:49.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:49 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3065350060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:49 smithi122 ceph-mon[100457]: Deploying daemon mds.h on smithi121 2023-08-25T02:24:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:49 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6836/2359799361,v1:172.21.15.122:6837/2359799361] up:boot 2023-08-25T02:24:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:49 smithi122 ceph-mon[100457]: fsmap 7 up:standby 2023-08-25T02:24:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:49 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:24:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:49 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3065350060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:49 smithi089 ceph-mon[98732]: Deploying daemon mds.h on smithi121 2023-08-25T02:24:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:49 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6836/2359799361,v1:172.21.15.122:6837/2359799361] up:boot 2023-08-25T02:24:49.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:49 smithi089 ceph-mon[98732]: fsmap 7 up:standby 2023-08-25T02:24:49.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:49 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:24:49.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:49 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3065350060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:50.219 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:49 smithi121 systemd[1]: Starting Ceph mds.h for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:50.544 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:50 smithi121 podman[124916]: 2023-08-25T02:24:50.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:50 smithi121 ceph-mon[100099]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:50 smithi122 ceph-mon[100457]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:50.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:50 smithi089 ceph-mon[98732]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:50.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:50.968 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:50 smithi121 ceph-mds[124956]: -- 172.21.15.121:0/506501256 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601df9671e0 con 0x5601df93d000 2023-08-25T02:24:50.969 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:50 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: starting mds.h at 2023-08-25T02:24:50.969 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:50 smithi121 bash[124916]: cb8e5d064163da52d272e34ec58423cada8c809d6d3be3c51a22f9d20b81f0e8 2023-08-25T02:24:50.969 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:24:50 smithi121 systemd[1]: Started Ceph mds.h for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.014+0000 7f9098373700 1 -- 172.21.15.89:0/1365907102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 msgr2=0x7f9090108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.014+0000 7f9098373700 1 --2- 172.21.15.89:0/1365907102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f9090108d60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f908800b0d0 tx=0x7f908800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 -- 172.21.15.89:0/1365907102 shutdown_connections 2023-08-25T02:24:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 --2- 172.21.15.89:0/1365907102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f9090108d60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 --2- 172.21.15.89:0/1365907102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 0x7f9090100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 --2- 172.21.15.89:0/1365907102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 0x7f9090101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 -- 172.21.15.89:0/1365907102 >> 172.21.15.89:0/1365907102 conn(0x7f90900fbe70 msgr2=0x7f90900fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 -- 172.21.15.89:0/1365907102 shutdown_connections 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 -- 172.21.15.89:0/1365907102 wait complete. 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 Processor -- start 2023-08-25T02:24:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.015+0000 7f9098373700 1 -- start start 2023-08-25T02:24:51.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 0x7f9090196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:51.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 0x7f9090196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:51.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909019e200 con 0x7f90901024b0 2023-08-25T02:24:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909019e340 con 0x7f9090101b10 2023-08-25T02:24:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909019e4d0 con 0x7f9090100920 2023-08-25T02:24:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46496/0 (socket says 172.21.15.89:46496) 2023-08-25T02:24:51.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 -- 172.21.15.89:0/1353700369 learned_addr learned my addr 172.21.15.89:0/1353700369 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:51.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f909610f700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 0x7f9090196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:51.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f909590e700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 0x7f9090196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:51.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 -- 172.21.15.89:0/1353700369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 msgr2=0x7f9090196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:51.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 0x7f9090196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 -- 172.21.15.89:0/1353700369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 msgr2=0x7f9090196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:51.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 0x7f9090196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 -- 172.21.15.89:0/1353700369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088009d00 con 0x7f90901024b0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9096910700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9088007a00 tx=0x7f90880094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800e040 con 0x7f90901024b0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9088003e10 con 0x7f90901024b0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909019e750 con 0x7f90901024b0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.016+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908801b9e0 con 0x7f90901024b0 2023-08-25T02:24:51.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.017+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909019eb70 con 0x7f90901024b0 2023-08-25T02:24:51.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.017+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909010c140 con 0x7f90901024b0 2023-08-25T02:24:51.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.019+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f9088005630 con 0x7f90901024b0 2023-08-25T02:24:51.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.020+0000 7f90877fe700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 0x7f907c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:51.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.020+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90880ccc60 con 0x7f90901024b0 2023-08-25T02:24:51.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.020+0000 7f909610f700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 0x7f907c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:51.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.020+0000 7f909610f700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 0x7f907c0a2460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9090101970 tx=0x7f908c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:51.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.023+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f9088096e80 con 0x7f90901024b0 2023-08-25T02:24:51.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.255+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9090107730 con 0x7f90901024b0 2023-08-25T02:24:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.256+0000 7f90877fe700 1 -- 172.21.15.89:0/1353700369 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7f908809a4a0 con 0x7f90901024b0 2023-08-25T02:24:51.258 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:51.258 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2023-08-25T02:24:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 msgr2=0x7f907c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 0x7f907c0a2460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9090101970 tx=0x7f908c007400 comp rx=0 tx=0).stop 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 msgr2=0x7f909019dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9088007a00 tx=0x7f90880094c0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 shutdown_connections 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90901024b0 0x7f909019dc30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9090100920 0x7f9090196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f907c09ffa0 0x7f907c0a2460 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 --2- 172.21.15.89:0/1353700369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9090101b10 0x7f9090196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.258+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 >> 172.21.15.89:0/1353700369 conn(0x7f90900fbe70 msgr2=0x7f90900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.259+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 shutdown_connections 2023-08-25T02:24:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:51.259+0000 7f9098373700 1 -- 172.21.15.89:0/1353700369 wait complete. 2023-08-25T02:24:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-08-25T02:24:51.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:51.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:51.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: Deploying daemon mds.i on smithi122 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1353700369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:boot 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: fsmap 8 up:standby 2023-08-25T02:24:51.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:51 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:24:51.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:51.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:51.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: Deploying daemon mds.i on smithi122 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1353700369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:boot 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: fsmap 8 up:standby 2023-08-25T02:24:51.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:51 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:24:52.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:52.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: Deploying daemon mds.i on smithi122 2023-08-25T02:24:52.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1353700369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:52.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:52.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:boot 2023-08-25T02:24:52.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: fsmap 8 up:standby 2023-08-25T02:24:52.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:51 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:24:52.478 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 systemd[1]: Starting Ceph mds.i for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:52.768 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 podman[126585]: 2023-08-25T02:24:52.768 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 ceph-mds[126625]: -- 172.21.15.122:0/3576186670 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55923ce0d1e0 con 0x55923cde2c00 2023-08-25T02:24:52.768 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 bash[126585]: 3ed57c59e61eaa8ba41ad6b14e070f36bb3a5292300f00b028e15b0019a517d8 2023-08-25T02:24:52.768 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: starting mds.i at 2023-08-25T02:24:53.228 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:24:52 smithi122 systemd[1]: Started Ceph mds.i for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:53.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:53.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 -- 172.21.15.89:0/3182908527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 msgr2=0x7fa884108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 --2- 172.21.15.89:0/3182908527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa884108e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00b0a0 tx=0x7fa87c00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 -- 172.21.15.89:0/3182908527 shutdown_connections 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 --2- 172.21.15.89:0/3182908527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa884108e30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 --2- 172.21.15.89:0/3182908527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8840fec10 0x7fa8840ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 --2- 172.21.15.89:0/3182908527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa884101be0 0x7fa884102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 -- 172.21.15.89:0/3182908527 >> 172.21.15.89:0/3182908527 conn(0x7fa8840fa520 msgr2=0x7fa8840fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 -- 172.21.15.89:0/3182908527 shutdown_connections 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.400+0000 7fa88aaab700 1 -- 172.21.15.89:0/3182908527 wait complete. 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 Processor -- start 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 -- start start 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 0x7fa884196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 0x7fa884196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:53.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:53.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419e180 con 0x7fa884102580 2023-08-25T02:24:53.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419e2c0 con 0x7fa8840fec10 2023-08-25T02:24:53.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa88aaab700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419e450 con 0x7fa884101be0 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa889048700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa883fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 0x7fa884196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa889048700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46510/0 (socket says 172.21.15.89:46510) 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.401+0000 7fa889048700 1 -- 172.21.15.89:0/2230660692 learned_addr learned my addr 172.21.15.89:0/2230660692 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa888847700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 0x7fa884196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 -- 172.21.15.89:0/2230660692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 msgr2=0x7fa884196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:53.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 0x7fa884196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 -- 172.21.15.89:0/2230660692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 msgr2=0x7fa884196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 0x7fa884196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 -- 172.21.15.89:0/2230660692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c009d00 con 0x7fa884102580 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa888847700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 0x7fa884196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa883fff700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 0x7fa884196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa889048700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00b780 tx=0x7fa87c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c00e040 con 0x7fa884102580 2023-08-25T02:24:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88419e6d0 con 0x7fa884102580 2023-08-25T02:24:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa87c004c50 con 0x7fa884102580 2023-08-25T02:24:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0053b0 con 0x7fa884102580 2023-08-25T02:24:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.402+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88419eb00 con 0x7fa884102580 2023-08-25T02:24:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.403+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88410c150 con 0x7fa884102580 2023-08-25T02:24:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.405+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fa87c01d9c0 con 0x7fa884102580 2023-08-25T02:24:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.405+0000 7fa881ffb700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 0x7fa8740a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.406+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa87c0cc8e0 con 0x7fa884102580 2023-08-25T02:24:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.406+0000 7fa888847700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 0x7fa8740a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:53.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.406+0000 7fa888847700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 0x7fa8740a2460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa884068ae0 tx=0x7fa878009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:53.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.409+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa87c096b00 con 0x7fa884102580 2023-08-25T02:24:53.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.681+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8841077f0 con 0x7fa884102580 2023-08-25T02:24:53.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.682+0000 7fa881ffb700 1 -- 172.21.15.89:0/2230660692 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7670 (secure 0 0 0) 0x7fa87c09a120 con 0x7fa884102580 2023-08-25T02:24:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:53.683 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2023-08-25T02:24:53.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 msgr2=0x7fa8740a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:53.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 0x7fa8740a2460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa884068ae0 tx=0x7fa878009260 comp rx=0 tx=0).stop 2023-08-25T02:24:53.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 msgr2=0x7fa88419dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:53.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00b780 tx=0x7fa87c00bab0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 shutdown_connections 2023-08-25T02:24:53.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa87409ffa0 0x7fa8740a2460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8840fec10 0x7fa884196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa884102580 0x7fa88419dbb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 --2- 172.21.15.89:0/2230660692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa884101be0 0x7fa884196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:53.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 >> 172.21.15.89:0/2230660692 conn(0x7fa8840fa520 msgr2=0x7fa8840fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:53.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 shutdown_connections 2023-08-25T02:24:53.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:53.684+0000 7fa88aaab700 1 -- 172.21.15.89:0/2230660692 wait complete. 2023-08-25T02:24:53.687 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 10 2023-08-25T02:24:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:boot 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: fsmap 9 up:standby 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:53.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:53.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:53.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: Deploying daemon mds.g on smithi089 2023-08-25T02:24:53.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:53.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:53 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2230660692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:54.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:boot 2023-08-25T02:24:54.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: fsmap 9 up:standby 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:54.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: Deploying daemon mds.g on smithi089 2023-08-25T02:24:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:54.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:53 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2230660692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:54.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:boot 2023-08-25T02:24:54.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: fsmap 9 up:standby 2023-08-25T02:24:54.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: Deploying daemon mds.g on smithi089 2023-08-25T02:24:54.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:54.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:53 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2230660692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:54.579 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 systemd[1]: Starting Ceph mds.g for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:54.886 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 podman[132970]: 2023-08-25T02:24:55.329 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 ceph-mds[133011]: -- 172.21.15.89:0/2625751336 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a48b21d1e0 con 0x55a48b1f2800 2023-08-25T02:24:55.329 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-g[132986]: starting mds.g at 2023-08-25T02:24:55.330 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 bash[132970]: eb15087f785e45256117903a84353d45e9cf6fd39ded800f7ff14aa07b69dfd0 2023-08-25T02:24:55.330 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:24:54 smithi089 systemd[1]: Started Ceph mds.g for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:55.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 -- 172.21.15.89:0/606664215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100e20 msgr2=0x7f0374101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 --2- 172.21.15.89:0/606664215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100e20 0x7f0374101240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0364009990 tx=0x7f0364009ca0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 -- 172.21.15.89:0/606664215 shutdown_connections 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 --2- 172.21.15.89:0/606664215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100430 0x7f0374107cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 --2- 172.21.15.89:0/606664215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100e20 0x7f0374101240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 --2- 172.21.15.89:0/606664215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 0x7f03740ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 -- 172.21.15.89:0/606664215 >> 172.21.15.89:0/606664215 conn(0x7f03740fa520 msgr2=0x7f03740fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.831+0000 7f037a9e3700 1 -- 172.21.15.89:0/606664215 shutdown_connections 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.832+0000 7f037a9e3700 1 -- 172.21.15.89:0/606664215 wait complete. 2023-08-25T02:24:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.832+0000 7f037a9e3700 1 Processor -- start 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.832+0000 7f037a9e3700 1 -- start start 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 0x7f0374195520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 0x7f037419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037419d0a0 con 0x7f0374100430 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037419d1e0 con 0x7f03740ffa70 2023-08-25T02:24:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037419d370 con 0x7f0374100e20 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46536/0 (socket says 172.21.15.89:46536) 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 -- 172.21.15.89:0/178710773 learned_addr learned my addr 172.21.15.89:0/178710773 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0373fff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 0x7f0374195520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0378f80700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 0x7f037419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 -- 172.21.15.89:0/178710773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 msgr2=0x7f037419cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 0x7f037419cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 -- 172.21.15.89:0/178710773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 msgr2=0x7f0374195520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 0x7f0374195520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 -- 172.21.15.89:0/178710773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03640095f0 con 0x7f0374100430 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0378f80700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 0x7f037419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f036bfff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f035c00b550 tx=0x7f035c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c011830 con 0x7f0374100430 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740feb00 con 0x7f0374100430 2023-08-25T02:24:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035c011e50 con 0x7f0374100430 2023-08-25T02:24:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00f2e0 con 0x7f0374100430 2023-08-25T02:24:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.833+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03740ff090 con 0x7f0374100430 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.835+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0374106dd0 con 0x7f0374100430 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.836+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f035c010580 con 0x7f0374100430 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.836+0000 7f0371ffb700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 0x7f03600a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.836+0000 7f0373fff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 0x7f03600a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.836+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f035c0ca860 con 0x7f0374100430 2023-08-25T02:24:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.837+0000 7f0373fff700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 0x7f03600a2460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0364006010 tx=0x7f036400b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:55.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:55.841+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f035c094a70 con 0x7f0374100430 2023-08-25T02:24:56.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.083+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f037410b120 con 0x7f0374100430 2023-08-25T02:24:56.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.083+0000 7f0371ffb700 1 -- 172.21.15.89:0/178710773 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8462 (secure 0 0 0) 0x7f035c098090 con 0x7f0374100430 2023-08-25T02:24:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:56.085 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 msgr2=0x7f03600a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 0x7f03600a2460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0364006010 tx=0x7f036400b560 comp rx=0 tx=0).stop 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 msgr2=0x7f0374195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f035c00b550 tx=0x7f035c00b860 comp rx=0 tx=0).stop 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 shutdown_connections 2023-08-25T02:24:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f036009ffa0 0x7f03600a2460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03740ffa70 0x7f0374195520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0374100e20 0x7f037419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 --2- 172.21.15.89:0/178710773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374100430 0x7f0374195a60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 >> 172.21.15.89:0/178710773 conn(0x7f03740fa520 msgr2=0x7f0374103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 shutdown_connections 2023-08-25T02:24:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:56.086+0000 7f037a9e3700 1 -- 172.21.15.89:0/178710773 wait complete. 2023-08-25T02:24:56.089 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 11 2023-08-25T02:24:56.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.89:6838/3717526782,v1:172.21.15.89:6839/3717526782] up:boot 2023-08-25T02:24:56.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: fsmap 10 up:standby 2023-08-25T02:24:56.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:24:56.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: Deploying daemon mds.a on smithi089 2023-08-25T02:24:56.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:55 smithi121 ceph-mon[100099]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:56.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.89:6838/3717526782,v1:172.21.15.89:6839/3717526782] up:boot 2023-08-25T02:24:56.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: fsmap 10 up:standby 2023-08-25T02:24:56.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:24:56.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: Deploying daemon mds.a on smithi089 2023-08-25T02:24:56.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:55 smithi122 ceph-mon[100457]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:56.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.89:6838/3717526782,v1:172.21.15.89:6839/3717526782] up:boot 2023-08-25T02:24:56.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: fsmap 10 up:standby 2023-08-25T02:24:56.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:24:56.248 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:56.249 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:56.250 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:56.250 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:56.250 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: Deploying daemon mds.a on smithi089 2023-08-25T02:24:56.251 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:55 smithi089 ceph-mon[98732]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:56.580 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:56 smithi089 systemd[1]: Starting Ceph mds.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:57.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/178710773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:57.039 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:56 smithi089 podman[133356]: 2023-08-25T02:24:57.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:56 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/178710773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:57.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:56 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/178710773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:57.329 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:57 smithi089 ceph-mds[133396]: -- 172.21.15.89:0/341520284 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627b80cd1e0 con 0x5627b80a3000 2023-08-25T02:24:57.329 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-a[133372]: starting mds.a at 2023-08-25T02:24:57.330 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:57 smithi089 bash[133356]: 2a26d801e41e8bf071280ab1fd983b059fbb82be7b5e03f6c573abf6f1058af4 2023-08-25T02:24:57.330 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:24:57 smithi089 systemd[1]: Started Ceph mds.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:24:58.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:24:58.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 -- 172.21.15.89:0/3344806799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 msgr2=0x7f0c600ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:58.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 --2- 172.21.15.89:0/3344806799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c600ffc20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0c480099c0 tx=0x7f0c48009cd0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 -- 172.21.15.89:0/3344806799 shutdown_connections 2023-08-25T02:24:58.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 --2- 172.21.15.89:0/3344806799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c60101390 0x7f0c60107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 --2- 172.21.15.89:0/3344806799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c600ffc20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 --2- 172.21.15.89:0/3344806799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c601009f0 0x7f0c60100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 -- 172.21.15.89:0/3344806799 >> 172.21.15.89:0/3344806799 conn(0x7f0c600fad50 msgr2=0x7f0c600fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:58.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 -- 172.21.15.89:0/3344806799 shutdown_connections 2023-08-25T02:24:58.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.239+0000 7f0c663c8700 1 -- 172.21.15.89:0/3344806799 wait complete. 2023-08-25T02:24:58.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.240+0000 7f0c663c8700 1 Processor -- start 2023-08-25T02:24:58.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.240+0000 7f0c663c8700 1 -- start start 2023-08-25T02:24:58.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:58.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c601009f0 0x7f0c60196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:58.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c60101390 0x7f0c6019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:58.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6019e180 con 0x7f0c600ff800 2023-08-25T02:24:58.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6019e2c0 con 0x7f0c601009f0 2023-08-25T02:24:58.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6019e450 con 0x7f0c60101390 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46560/0 (socket says 172.21.15.89:46560) 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 -- 172.21.15.89:0/4281417041 learned_addr learned my addr 172.21.15.89:0/4281417041 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5f7fe700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c601009f0 0x7f0c60196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c64965700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c60101390 0x7f0c6019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 -- 172.21.15.89:0/4281417041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c60101390 msgr2=0x7f0c6019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c60101390 0x7f0c6019dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 -- 172.21.15.89:0/4281417041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c601009f0 msgr2=0x7f0c60196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c601009f0 0x7f0c60196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 -- 172.21.15.89:0/4281417041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c480095f0 con 0x7f0c600ff800 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5ffff700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009fd0 tx=0x7f0c48009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4801d070 con 0x7f0c600ff800 2023-08-25T02:24:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6019e6d0 con 0x7f0c600ff800 2023-08-25T02:24:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c48005030 con 0x7f0c600ff800 2023-08-25T02:24:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48004a90 con 0x7f0c600ff800 2023-08-25T02:24:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.241+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6019eaf0 con 0x7f0c600ff800 2023-08-25T02:24:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.242+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c60107980 con 0x7f0c600ff800 2023-08-25T02:24:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.243+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0c480051a0 con 0x7f0c600ff800 2023-08-25T02:24:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.244+0000 7f0c5d7fa700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 0x7f0c4c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:24:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.244+0000 7f0c5f7fe700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 0x7f0c4c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:24:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.244+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0c480cbab0 con 0x7f0c600ff800 2023-08-25T02:24:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.244+0000 7f0c5f7fe700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 0x7f0c4c0a2460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009c60 tx=0x7f0c50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:24:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.248+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0c48095d70 con 0x7f0c600ff800 2023-08-25T02:24:58.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: Deploying daemon mds.b on smithi121 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.89:6840/3065557297,v1:172.21.15.89:6841/3065557297] up:boot 2023-08-25T02:24:58.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: fsmap 11 up:standby 2023-08-25T02:24:58.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:58 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:24:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: Deploying daemon mds.b on smithi121 2023-08-25T02:24:58.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:58.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.89:6840/3065557297,v1:172.21.15.89:6841/3065557297] up:boot 2023-08-25T02:24:58.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: fsmap 11 up:standby 2023-08-25T02:24:58.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:58 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:24:58.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.481+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c6010b010 con 0x7f0c600ff800 2023-08-25T02:24:58.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.482+0000 7f0c5d7fa700 1 -- 172.21.15.89:0/4281417041 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7f0c48099390 con 0x7f0c600ff800 2023-08-25T02:24:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:24:58.483 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2023-08-25T02:24:58.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 msgr2=0x7f0c4c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:58.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 0x7f0c4c0a2460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009c60 tx=0x7f0c50009260 comp rx=0 tx=0).stop 2023-08-25T02:24:58.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 msgr2=0x7f0c60196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:24:58.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009fd0 tx=0x7f0c48009580 comp rx=0 tx=0).stop 2023-08-25T02:24:58.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 shutdown_connections 2023-08-25T02:24:58.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c60101390 0x7f0c6019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c600ff800 0x7f0c60196600 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0c4c09ffa0 0x7f0c4c0a2460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 --2- 172.21.15.89:0/4281417041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c601009f0 0x7f0c60196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:24:58.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 >> 172.21.15.89:0/4281417041 conn(0x7f0c600fad50 msgr2=0x7f0c600fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:24:58.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 shutdown_connections 2023-08-25T02:24:58.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:24:58.484+0000 7f0c663c8700 1 -- 172.21.15.89:0/4281417041 wait complete. 2023-08-25T02:24:58.488 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2023-08-25T02:24:58.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:24:58.496 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: Deploying daemon mds.b on smithi121 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.89:6840/3065557297,v1:172.21.15.89:6841/3065557297] up:boot 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: fsmap 11 up:standby 2023-08-25T02:24:58.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:58 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:24:58.761 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:58 smithi121 systemd[1]: Starting Ceph mds.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:24:59.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:24:59 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4281417041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:59.380 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:59 smithi121 podman[125255]: 2023-08-25T02:24:59.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:24:59 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4281417041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:59.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:24:59 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4281417041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:24:59.718 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:59 smithi121 ceph-mds[125295]: -- 172.21.15.121:0/281481860 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d67700f1e0 con 0x55d676fe4800 2023-08-25T02:24:59.718 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:59 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: starting mds.b at 2023-08-25T02:24:59.718 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:59 smithi121 bash[125255]: 8e673ec3147854478327fd6701ddc666c8a2da9f008774ca0f9967de818034c9 2023-08-25T02:24:59.719 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:24:59 smithi121 systemd[1]: Started Ceph mds.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:25:00.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:25:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/651523985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c8100920 msgr2=0x7fc2c8100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/651523985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c8100920 0x7fc2c8100d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0099c0 tx=0x7fc2bc009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/651523985 shutdown_connections 2023-08-25T02:25:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/651523985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c81024b0 0x7fc2c8108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/651523985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c8100920 0x7fc2c8100d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/651523985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 0x7fc2c8101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/651523985 >> 172.21.15.89:0/651523985 conn(0x7fc2c80fbe70 msgr2=0x7fc2c80fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/651523985 shutdown_connections 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.625+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/651523985 wait complete. 2023-08-25T02:25:00.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 Processor -- start 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 -- start start 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 0x7fc2c80746d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 0x7fc2c8072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80739f0 con 0x7fc2c81024b0 2023-08-25T02:25:00.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8073b60 con 0x7fc2c8100920 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2cd2c8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c81a0680 con 0x7fc2c8101b10 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2c77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2c77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46578/0 (socket says 172.21.15.89:46578) 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2c77fe700 1 -- 172.21.15.89:0/3287116940 learned_addr learned my addr 172.21.15.89:0/3287116940 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2c6ffd700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 0x7fc2c80746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.626+0000 7fc2c67fc700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 0x7fc2c8072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:00.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 -- 172.21.15.89:0/3287116940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 msgr2=0x7fc2c8072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 0x7fc2c8072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 -- 172.21.15.89:0/3287116940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 msgr2=0x7fc2c80746d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 0x7fc2c80746d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 -- 172.21.15.89:0/3287116940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0095f0 con 0x7fc2c81024b0 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c6ffd700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 0x7fc2c80746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2c77fe700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0007ee0 tx=0x7fc2c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:00.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000dd20 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2c000f040 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c81a0960 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00048a0 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.627+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c81a0e20 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.628+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c810c180 con 0x7fc2c81024b0 2023-08-25T02:25:00.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.629+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fc2c0003a00 con 0x7fc2c81024b0 2023-08-25T02:25:00.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.630+0000 7fc2affff700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 0x7fc2b00a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:00.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.630+0000 7fc2c6ffd700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 0x7fc2b00a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:00.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.630+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc2c00cadf0 con 0x7fc2c81024b0 2023-08-25T02:25:00.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.630+0000 7fc2c6ffd700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 0x7fc2b00a2460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009990 tx=0x7fc2bc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:00.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.633+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fc2c0095020 con 0x7fc2c81024b0 2023-08-25T02:25:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:00.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:boot 2023-08-25T02:25:00.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: fsmap 12 up:standby 2023-08-25T02:25:00.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:00 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:25:00.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.880+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2c8107860 con 0x7fc2c81024b0 2023-08-25T02:25:00.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.881+0000 7fc2affff700 1 -- 172.21.15.89:0/3287116940 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10049 (secure 0 0 0) 0x7fc2c0098640 con 0x7fc2c81024b0 2023-08-25T02:25:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:00.882 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2023-08-25T02:25:00.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 msgr2=0x7fc2b00a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:00.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 0x7fc2b00a2460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009990 tx=0x7fc2bc01a040 comp rx=0 tx=0).stop 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 msgr2=0x7fc2c8073250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0007ee0 tx=0x7fc2c000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 shutdown_connections 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc2b009ffa0 0x7fc2b00a2460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2c8100920 0x7fc2c80746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c81024b0 0x7fc2c8073250 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 --2- 172.21.15.89:0/3287116940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2c8101b10 0x7fc2c8072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:00.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 >> 172.21.15.89:0/3287116940 conn(0x7fc2c80fbe70 msgr2=0x7fc2c80fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:00.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 shutdown_connections 2023-08-25T02:25:00.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:00.883+0000 7fc2cd2c8700 1 -- 172.21.15.89:0/3287116940 wait complete. 2023-08-25T02:25:00.886 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 13 2023-08-25T02:25:00.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:00.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:00.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:boot 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: fsmap 12 up:standby 2023-08-25T02:25:00.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:00 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:25:00.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:00.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:00.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:boot 2023-08-25T02:25:00.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: fsmap 12 up:standby 2023-08-25T02:25:00.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:00 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:25:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 -- 172.21.15.89:0/1149356015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 msgr2=0x7f6a20107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1149356015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a20107c50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800b0d0 tx=0x7f6a1800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 -- 172.21.15.89:0/1149356015 shutdown_connections 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1149356015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a200ff810 0x7f6a200ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1149356015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a20107c50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1149356015 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a20100a00 0x7f6a20100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.024+0000 7f6a267d4700 1 -- 172.21.15.89:0/1149356015 >> 172.21.15.89:0/1149356015 conn(0x7f6a200fad40 msgr2=0x7f6a200fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- 172.21.15.89:0/1149356015 shutdown_connections 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- 172.21.15.89:0/1149356015 wait complete. 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 Processor -- start 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- start start 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 0x7f6a20195580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a20100a00 0x7f6a20195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2019d100 con 0x7f6a201013a0 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2019d240 con 0x7f6a20100a00 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.025+0000 7f6a267d4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2019d3d0 con 0x7f6a200ff810 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 0x7f6a20195580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46592/0 (socket says 172.21.15.89:46592) 2023-08-25T02:25:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 0x7f6a20195580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:38524/0 (socket says 172.21.15.89:38524) 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 -- 172.21.15.89:0/1260438376 learned_addr learned my addr 172.21.15.89:0/1260438376 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1f7fe700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a20100a00 0x7f6a20195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 -- 172.21.15.89:0/1260438376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 msgr2=0x7f6a20195580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 0x7f6a20195580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 -- 172.21.15.89:0/1260438376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a20100a00 msgr2=0x7f6a20195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a20100a00 0x7f6a20195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 -- 172.21.15.89:0/1260438376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009d00 con 0x7f6a201013a0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a24d71700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a18008f10 tx=0x7f6a18008f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800e040 con 0x7f6a201013a0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2019d650 con 0x7f6a201013a0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a1800be60 con 0x7f6a201013a0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801b6a0 con 0x7f6a201013a0 2023-08-25T02:25:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.026+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2019da80 con 0x7f6a201013a0 2023-08-25T02:25:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.027+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20106cb0 con 0x7f6a201013a0 2023-08-25T02:25:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.028+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6a18005090 con 0x7f6a201013a0 2023-08-25T02:25:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.029+0000 7f6a1d7fa700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 0x7f6a080a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.029+0000 7f6a1ffff700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 0x7f6a080a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.029+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6a180cc830 con 0x7f6a201013a0 2023-08-25T02:25:01.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.029+0000 7f6a1ffff700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 0x7f6a080a24e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a20100860 tx=0x7f6a10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.032+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6a18096a50 con 0x7f6a201013a0 2023-08-25T02:25:01.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.236+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2002ce20 con 0x7f6a201013a0 2023-08-25T02:25:01.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.237+0000 7f6a1d7fa700 1 -- 172.21.15.89:0/1260438376 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f6a1809a070 con 0x7f6a201013a0 2023-08-25T02:25:01.239 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:01.239 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:24:24.361562+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:01.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 msgr2=0x7f6a080a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 0x7f6a080a24e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a20100860 tx=0x7f6a10009260 comp rx=0 tx=0).stop 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 msgr2=0x7f6a2019cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a18008f10 tx=0x7f6a18008f40 comp rx=0 tx=0).stop 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 shutdown_connections 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a201013a0 0x7f6a2019cb30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a200ff810 0x7f6a20195580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6a080a0020 0x7f6a080a24e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 --2- 172.21.15.89:0/1260438376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a20100a00 0x7f6a20195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 >> 172.21.15.89:0/1260438376 conn(0x7f6a200fad40 msgr2=0x7f6a20103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 shutdown_connections 2023-08-25T02:25:01.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.240+0000 7f6a267d4700 1 -- 172.21.15.89:0/1260438376 wait complete. 2023-08-25T02:25:01.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:25:01.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:25:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 -- 172.21.15.89:0/2228255361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94101b30 msgr2=0x7f3d94101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/2228255361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94101b30 0x7f3d94101f90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3d840099c0 tx=0x7f3d84009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 -- 172.21.15.89:0/2228255361 shutdown_connections 2023-08-25T02:25:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/2228255361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 0x7f3d94108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/2228255361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94100940 0x7f3d94100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/2228255361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94101b30 0x7f3d94101f90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 -- 172.21.15.89:0/2228255361 >> 172.21.15.89:0/2228255361 conn(0x7f3d940fbe90 msgr2=0x7f3d940fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 -- 172.21.15.89:0/2228255361 shutdown_connections 2023-08-25T02:25:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.381+0000 7f3d9be5b700 1 -- 172.21.15.89:0/2228255361 wait complete. 2023-08-25T02:25:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 Processor -- start 2023-08-25T02:25:01.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 -- start start 2023-08-25T02:25:01.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94101b30 0x7f3d94072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 0x7f3d94073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940739f0 con 0x7f3d94100940 2023-08-25T02:25:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94073b60 con 0x7f3d941024d0 2023-08-25T02:25:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9be5b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d941a0690 con 0x7f3d94101b30 2023-08-25T02:25:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46602/0 (socket says 172.21.15.89:46602) 2023-08-25T02:25:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 -- 172.21.15.89:0/1437861709 learned_addr learned my addr 172.21.15.89:0/1437861709 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d993f6700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94101b30 0x7f3d94072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d9a3f8700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 0x7f3d94073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 -- 172.21.15.89:0/1437861709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94101b30 msgr2=0x7f3d94072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94101b30 0x7f3d94072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 -- 172.21.15.89:0/1437861709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 msgr2=0x7f3d94073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 0x7f3d94073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 -- 172.21.15.89:0/1437861709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840095f0 con 0x7f3d94100940 2023-08-25T02:25:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d99bf7700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00efc0 tx=0x7f3d8c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.382+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c018070 con 0x7f3d94100940 2023-08-25T02:25:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.383+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d941a0910 con 0x7f3d94100940 2023-08-25T02:25:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.383+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d8c00f040 con 0x7f3d94100940 2023-08-25T02:25:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.383+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c004280 con 0x7f3d94100940 2023-08-25T02:25:01.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.383+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d941a0e70 con 0x7f3d94100940 2023-08-25T02:25:01.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.384+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d94107ed0 con 0x7f3d94100940 2023-08-25T02:25:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.384+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f3d8c003a00 con 0x7f3d94100940 2023-08-25T02:25:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.385+0000 7f3d8affd700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 0x7f3d800a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.385+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d8c0cac40 con 0x7f3d94100940 2023-08-25T02:25:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.385+0000 7f3d993f6700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 0x7f3d800a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.385+0000 7f3d993f6700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 0x7f3d800a2410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400b580 tx=0x7f3d84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.389+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f3d8c094f00 con 0x7f3d94100940 2023-08-25T02:25:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:01 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3287116940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:01 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1260438376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:01.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.593+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d9410c1a0 con 0x7f3d94100940 2023-08-25T02:25:01.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.594+0000 7f3d8affd700 1 -- 172.21.15.89:0/1437861709 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d9410c1a0 con 0x7f3d94100940 2023-08-25T02:25:01.595 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:25:01.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 msgr2=0x7f3d800a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 0x7f3d800a2410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400b580 tx=0x7f3d84005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 msgr2=0x7f3d940746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00efc0 tx=0x7f3d8c00c3a0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 shutdown_connections 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f3d8009ff50 0x7f3d800a2410 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d941024d0 0x7f3d94073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d94100940 0x7f3d940746d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 --2- 172.21.15.89:0/1437861709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d94101b30 0x7f3d94072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 >> 172.21.15.89:0/1437861709 conn(0x7f3d940fbe90 msgr2=0x7f3d94105010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 shutdown_connections 2023-08-25T02:25:01.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.596+0000 7f3d9be5b700 1 -- 172.21.15.89:0/1437861709 wait complete. 2023-08-25T02:25:01.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-08-25T02:25:01.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 -- 172.21.15.89:0/184713245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc101380 msgr2=0x7fccdc103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 --2- 172.21.15.89:0/184713245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc101380 0x7fccdc103770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009990 tx=0x7fccd8009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 -- 172.21.15.89:0/184713245 shutdown_connections 2023-08-25T02:25:01.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 --2- 172.21.15.89:0/184713245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc1065e0 0x7fccdc108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 --2- 172.21.15.89:0/184713245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc101380 0x7fccdc103770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 --2- 172.21.15.89:0/184713245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc103cb0 0x7fccdc1060a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 -- 172.21.15.89:0/184713245 >> 172.21.15.89:0/184713245 conn(0x7fccdc0fad50 msgr2=0x7fccdc0fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 -- 172.21.15.89:0/184713245 shutdown_connections 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.733+0000 7fcce372e700 1 -- 172.21.15.89:0/184713245 wait complete. 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 Processor -- start 2023-08-25T02:25:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 -- start start 2023-08-25T02:25:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc101380 0x7fccdc196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc1065e0 0x7fccdc19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc19e160 con 0x7fccdc103cb0 2023-08-25T02:25:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc19e2a0 con 0x7fccdc101380 2023-08-25T02:25:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce372e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc19e430 con 0x7fccdc1065e0 2023-08-25T02:25:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce0cc9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce0cc9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46618/0 (socket says 172.21.15.89:46618) 2023-08-25T02:25:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce0cc9700 1 -- 172.21.15.89:0/3056815420 learned_addr learned my addr 172.21.15.89:0/3056815420 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce14ca700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc101380 0x7fccdc196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.734+0000 7fcce1ccb700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc1065e0 0x7fccdc19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 -- 172.21.15.89:0/3056815420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc1065e0 msgr2=0x7fccdc19db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc1065e0 0x7fccdc19db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 -- 172.21.15.89:0/3056815420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc101380 msgr2=0x7fccdc196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc101380 0x7fccdc196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 -- 172.21.15.89:0/3056815420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd80095f0 con 0x7fccdc103cb0 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce0cc9700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00e9c0 tx=0x7fcccc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0097b0 con 0x7fccdc103cb0 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcccc00ee60 con 0x7fccdc103cb0 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc19e710 con 0x7fccdc103cb0 2023-08-25T02:25:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0054f0 con 0x7fccdc103cb0 2023-08-25T02:25:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.735+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc19ebd0 con 0x7fccdc103cb0 2023-08-25T02:25:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.736+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc10c100 con 0x7fccdc103cb0 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.737+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fcccc004750 con 0x7fccdc103cb0 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.738+0000 7fccd27fc700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 0x7fccc80a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.738+0000 7fcce14ca700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 0x7fccc80a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.738+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcccc014070 con 0x7fccdc103cb0 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.738+0000 7fcce14ca700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 0x7fccc80a2460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b2e0 tx=0x7fccd8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.741+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fcccc0953c0 con 0x7fccdc103cb0 2023-08-25T02:25:01.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.942+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fccdc10c370 con 0x7fccdc103cb0 2023-08-25T02:25:01.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.943+0000 7fccd27fc700 1 -- 172.21.15.89:0/3056815420 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcccc0989e0 con 0x7fccdc103cb0 2023-08-25T02:25:01.948 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:25:01.949 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-08-25T02:25:01.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 msgr2=0x7fccc80a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 0x7fccc80a2460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b2e0 tx=0x7fccd8005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 msgr2=0x7fccdc196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:01.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00e9c0 tx=0x7fcccc00ecd0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 shutdown_connections 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fccc809ffa0 0x7fccc80a2460 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccdc101380 0x7fccdc196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccdc1065e0 0x7fccdc19db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 --2- 172.21.15.89:0/3056815420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccdc103cb0 0x7fccdc196b70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 >> 172.21.15.89:0/3056815420 conn(0x7fccdc0fad50 msgr2=0x7fccdc1027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 shutdown_connections 2023-08-25T02:25:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:01.945+0000 7fcce372e700 1 -- 172.21.15.89:0/3056815420 wait complete. 2023-08-25T02:25:01.958 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-08-25T02:25:01.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-08-25T02:25:01.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:01 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3287116940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:01.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:01 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1260438376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:01 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3287116940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:01 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1260438376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 -- 172.21.15.89:0/516647940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb44101f20 msgr2=0x7fcb44102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 --2- 172.21.15.89:0/516647940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb44101f20 0x7fcb44102340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb34009990 tx=0x7fcb34009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 -- 172.21.15.89:0/516647940 shutdown_connections 2023-08-25T02:25:02.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 --2- 172.21.15.89:0/516647940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb440ff330 0x7fcb44108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 --2- 172.21.15.89:0/516647940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb44101f20 0x7fcb44102340 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 --2- 172.21.15.89:0/516647940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440fe990 0x7fcb440fedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 -- 172.21.15.89:0/516647940 >> 172.21.15.89:0/516647940 conn(0x7fcb440fa4f0 msgr2=0x7fcb440fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:02.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 -- 172.21.15.89:0/516647940 shutdown_connections 2023-08-25T02:25:02.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.092+0000 7fcb4a211700 1 -- 172.21.15.89:0/516647940 wait complete. 2023-08-25T02:25:02.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 Processor -- start 2023-08-25T02:25:02.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 -- start start 2023-08-25T02:25:02.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440ff330 0x7fcb44196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb44101f20 0x7fcb4419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4419e220 con 0x7fcb440fe990 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4419e360 con 0x7fcb44101f20 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb4a211700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4419e4f0 con 0x7fcb440ff330 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46630/0 (socket says 172.21.15.89:46630) 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb437fe700 1 -- 172.21.15.89:0/1307918417 learned_addr learned my addr 172.21.15.89:0/1307918417 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.093+0000 7fcb43fff700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb44101f20 0x7fcb4419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb42ffd700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440ff330 0x7fcb44196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 -- 172.21.15.89:0/1307918417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440ff330 msgr2=0x7fcb44196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440ff330 0x7fcb44196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 -- 172.21.15.89:0/1307918417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb44101f20 msgr2=0x7fcb4419dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb44101f20 0x7fcb4419dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 -- 172.21.15.89:0/1307918417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340095f0 con 0x7fcb440fe990 2023-08-25T02:25:02.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb437fe700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb340098e0 tx=0x7fcb3400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3401d070 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb34022470 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4419e770 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34017650 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.094+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4419eb90 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.095+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4410c170 con 0x7fcb440fe990 2023-08-25T02:25:02.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.096+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fcb340225e0 con 0x7fcb440fe990 2023-08-25T02:25:02.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.097+0000 7fcb40ff9700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 0x7fcb2c0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.097+0000 7fcb42ffd700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 0x7fcb2c0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.097+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcb340cc190 con 0x7fcb440fe990 2023-08-25T02:25:02.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.097+0000 7fcb42ffd700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 0x7fcb2c0a2490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcb38006fd0 tx=0x7fcb38008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:02.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.101+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fcb340963b0 con 0x7fcb440fe990 2023-08-25T02:25:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.303+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcb4410c380 con 0x7fcb440fe990 2023-08-25T02:25:02.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.595+0000 7fcb40ff9700 1 -- 172.21.15.89:0/1307918417 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcb340999d0 con 0x7fcb440fe990 2023-08-25T02:25:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 msgr2=0x7fcb2c0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 0x7fcb2c0a2490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcb38006fd0 tx=0x7fcb38008040 comp rx=0 tx=0).stop 2023-08-25T02:25:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 msgr2=0x7fcb441966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb340098e0 tx=0x7fcb3400f690 comp rx=0 tx=0).stop 2023-08-25T02:25:02.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 shutdown_connections 2023-08-25T02:25:02.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fcb2c09ffd0 0x7fcb2c0a2490 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb44101f20 0x7fcb4419dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb440fe990 0x7fcb441966a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 --2- 172.21.15.89:0/1307918417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb440ff330 0x7fcb44196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.598+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 >> 172.21.15.89:0/1307918417 conn(0x7fcb440fa4f0 msgr2=0x7fcb440fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:02.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.600+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 shutdown_connections 2023-08-25T02:25:02.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.600+0000 7fcb4a211700 1 -- 172.21.15.89:0/1307918417 wait complete. 2023-08-25T02:25:02.603 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-08-25T02:25:02.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-08-25T02:25:02.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 -- 172.21.15.89:0/879334301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d18101390 msgr2=0x7f1d18103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 --2- 172.21.15.89:0/879334301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d18101390 0x7f1d18103780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1d14009990 tx=0x7f1d14009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 -- 172.21.15.89:0/879334301 shutdown_connections 2023-08-25T02:25:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 --2- 172.21.15.89:0/879334301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d181065f0 0x7f1d18108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 --2- 172.21.15.89:0/879334301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d18101390 0x7f1d18103780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 --2- 172.21.15.89:0/879334301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18103cc0 0x7f1d181060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 -- 172.21.15.89:0/879334301 >> 172.21.15.89:0/879334301 conn(0x7f1d180fad80 msgr2=0x7f1d180fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 -- 172.21.15.89:0/879334301 shutdown_connections 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.746+0000 7f1d201d3700 1 -- 172.21.15.89:0/879334301 wait complete. 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 Processor -- start 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 -- start start 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 0x7f1d18196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 0x7f1d18196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1819e1c0 con 0x7f1d181065f0 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1819e300 con 0x7f1d18103cc0 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d201d3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1819e490 con 0x7f1d18101390 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1df6f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 0x7f1d18196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46648/0 (socket says 172.21.15.89:46648) 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 -- 172.21.15.89:0/1148929684 learned_addr learned my addr 172.21.15.89:0/1148929684 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1d76e700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 0x7f1d18196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 -- 172.21.15.89:0/1148929684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 msgr2=0x7f1d18196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 0x7f1d18196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 -- 172.21.15.89:0/1148929684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 msgr2=0x7f1d18196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 0x7f1d18196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1e770700 1 -- 172.21.15.89:0/1148929684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d140095f0 con 0x7f1d181065f0 2023-08-25T02:25:02.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1d76e700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 0x7f1d18196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.747+0000 7f1d1df6f700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 0x7f1d18196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d1e770700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d10007ee0 tx=0x7f1d1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1000dd20 con 0x7f1d181065f0 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d1000f040 con 0x7f1d181065f0 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1819e770 con 0x7f1d181065f0 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d100048a0 con 0x7f1d181065f0 2023-08-25T02:25:02.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.748+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1819ec30 con 0x7f1d181065f0 2023-08-25T02:25:02.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.749+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d180fc900 con 0x7f1d181065f0 2023-08-25T02:25:02.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.750+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f1d10003a00 con 0x7f1d181065f0 2023-08-25T02:25:02.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.751+0000 7f1d0effd700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 0x7f1d040a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:02.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.751+0000 7f1d1df6f700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 0x7f1d040a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:02.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.751+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f1d100caf60 con 0x7f1d181065f0 2023-08-25T02:25:02.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.751+0000 7f1d1df6f700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 0x7f1d040a2410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b2e0 tx=0x7f1d14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:02.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.755+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1d10095020 con 0x7f1d181065f0 2023-08-25T02:25:02.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:02 smithi089 ceph-mon[98732]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:02 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1437861709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:02 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3056815420' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-08-25T02:25:02.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:02.963+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1d1810c0f0 con 0x7f1d181065f0 2023-08-25T02:25:02.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:02 smithi121 ceph-mon[100099]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:02.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:02 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1437861709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:02.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:02 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3056815420' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-08-25T02:25:02.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:02 smithi122 ceph-mon[100457]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:02.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:02 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1437861709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:02.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:02 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3056815420' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-08-25T02:25:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.605+0000 7f1d0effd700 1 -- 172.21.15.89:0/1148929684 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1d10098640 con 0x7f1d181065f0 2023-08-25T02:25:03.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 msgr2=0x7f1d040a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 0x7f1d040a2410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b2e0 tx=0x7f1d14005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 msgr2=0x7f1d1819dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d10007ee0 tx=0x7f1d1000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 shutdown_connections 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d181065f0 0x7f1d1819dbf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d18101390 0x7f1d18196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1d0409ff50 0x7f1d040a2410 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 --2- 172.21.15.89:0/1148929684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18103cc0 0x7f1d18196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.616+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 >> 172.21.15.89:0/1148929684 conn(0x7f1d180fad80 msgr2=0x7f1d18102000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.617+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 shutdown_connections 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.617+0000 7f1d201d3700 1 -- 172.21.15.89:0/1148929684 wait complete. 2023-08-25T02:25:03.619 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-08-25T02:25:03.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-08-25T02:25:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 -- 172.21.15.89:0/4160222247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc0ff820 msgr2=0x7f2bdc0ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 --2- 172.21.15.89:0/4160222247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc0ffc40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0099c0 tx=0x7f2bcc009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 -- 172.21.15.89:0/4160222247 shutdown_connections 2023-08-25T02:25:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 --2- 172.21.15.89:0/4160222247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 0x7f2bdc107c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 --2- 172.21.15.89:0/4160222247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc0ffc40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 --2- 172.21.15.89:0/4160222247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc100a10 0x7f2bdc100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 -- 172.21.15.89:0/4160222247 >> 172.21.15.89:0/4160222247 conn(0x7f2bdc0fadb0 msgr2=0x7f2bdc0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 -- 172.21.15.89:0/4160222247 shutdown_connections 2023-08-25T02:25:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.762+0000 7f2be47e2700 1 -- 172.21.15.89:0/4160222247 wait complete. 2023-08-25T02:25:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 Processor -- start 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 -- start start 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 0x7f2bdc19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc19e1f0 con 0x7f2bdc100a10 2023-08-25T02:25:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc19e360 con 0x7f2bdc0ff820 2023-08-25T02:25:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be47e2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc19e520 con 0x7f2bdc1013b0 2023-08-25T02:25:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46664/0 (socket says 172.21.15.89:46664) 2023-08-25T02:25:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 -- 172.21.15.89:0/28562118 learned_addr learned my addr 172.21.15.89:0/28562118 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be2d7f700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 0x7f2bdc19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 -- 172.21.15.89:0/28562118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 msgr2=0x7f2bdc19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 0x7f2bdc19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be257e700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 -- 172.21.15.89:0/28562118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 msgr2=0x7f2bdc069620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc069620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 -- 172.21.15.89:0/28562118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc0095f0 con 0x7f2bdc100a10 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be257e700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.763+0000 7f2be1d7d700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800b530 tx=0x7f2bd800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.764+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8011830 con 0x7f2bdc100a10 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.764+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bd8011e50 con 0x7f2bdc100a10 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.764+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc19e800 con 0x7f2bdc100a10 2023-08-25T02:25:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.764+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800f2e0 con 0x7f2bdc100a10 2023-08-25T02:25:03.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.764+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc19ed10 con 0x7f2bdc100a10 2023-08-25T02:25:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.765+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc106e30 con 0x7f2bdc100a10 2023-08-25T02:25:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.766+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f2bd8010380 con 0x7f2bdc100a10 2023-08-25T02:25:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.767+0000 7f2bd37fe700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 0x7f2bc80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.767+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f2bd80cb1e0 con 0x7f2bdc100a10 2023-08-25T02:25:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.767+0000 7f2be257e700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 0x7f2bc80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.767+0000 7f2be257e700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 0x7f2bc80a2410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:03.771+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f2bd8095080 con 0x7f2bdc100a10 2023-08-25T02:25:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: osdmap e77: 12 total, 12 up, 12 in 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-08-25T02:25:03.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: osdmap e77: 12 total, 12 up, 12 in 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-08-25T02:25:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1307918417' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: osdmap e77: 12 total, 12 up, 12 in 2023-08-25T02:25:03.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-08-25T02:25:03.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:03.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:03.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:03.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:25:04.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.022+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2bdc10b300 con 0x7f2bdc100a10 2023-08-25T02:25:04.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.601+0000 7f2bd37fe700 1 -- 172.21.15.89:0/28562118 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2bd80986a0 con 0x7f2bdc100a10 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: pgmap v247: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: osdmap e78: 12 total, 12 up, 12 in 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-08-25T02:25:04.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-08-25T02:25:04.606 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:04 smithi121 ceph-mon[100099]: osdmap e79: 12 total, 12 up, 12 in 2023-08-25T02:25:04.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.604+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 msgr2=0x7f2bc80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.604+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 0x7f2bc80a2410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc000bc0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.604+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 msgr2=0x7f2bdc19b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.604+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800b530 tx=0x7f2bd800b840 comp rx=0 tx=0).stop 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 shutdown_connections 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f2bc809ff50 0x7f2bc80a2410 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc000bc0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ff820 0x7f2bdc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bdc1013b0 0x7f2bdc19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 --2- 172.21.15.89:0/28562118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bdc100a10 0x7f2bdc19b3d0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800b530 tx=0x7f2bd800b840 comp rx=0 tx=0).stop 2023-08-25T02:25:04.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.605+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 >> 172.21.15.89:0/28562118 conn(0x7f2bdc0fadb0 msgr2=0x7f2bdc103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:04.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.606+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 shutdown_connections 2023-08-25T02:25:04.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.606+0000 7f2be47e2700 1 -- 172.21.15.89:0/28562118 wait complete. 2023-08-25T02:25:04.609 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 2 and data pool 3 2023-08-25T02:25:04.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.749+0000 7f521fee6700 1 -- 172.21.15.89:0/18397089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 msgr2=0x7f5218100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.749+0000 7f521fee6700 1 --2- 172.21.15.89:0/18397089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f5218100e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f52080099c0 tx=0x7f5208009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.749+0000 7f521fee6700 1 -- 172.21.15.89:0/18397089 shutdown_connections 2023-08-25T02:25:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.749+0000 7f521fee6700 1 --2- 172.21.15.89:0/18397089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52180ff800 0x7f52180ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.749+0000 7f521fee6700 1 --2- 172.21.15.89:0/18397089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5218101390 0x7f5218107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 --2- 172.21.15.89:0/18397089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f5218100e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- 172.21.15.89:0/18397089 >> 172.21.15.89:0/18397089 conn(0x7f52180fad50 msgr2=0x7f52180fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- 172.21.15.89:0/18397089 shutdown_connections 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- 172.21.15.89:0/18397089 wait complete. 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 Processor -- start 2023-08-25T02:25:04.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- start start 2023-08-25T02:25:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218101390 0x7f521819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521819d030 con 0x7f52181009f0 2023-08-25T02:25:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521819d170 con 0x7f5218101390 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521fee6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521819d300 con 0x7f52180ff800 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521d481700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521dc82700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521dc82700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:38616/0 (socket says 172.21.15.89:38616) 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521e483700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218101390 0x7f521819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521dc82700 1 -- 172.21.15.89:0/1681095709 learned_addr learned my addr 172.21.15.89:0/1681095709 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:04.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.750+0000 7f521d481700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46682/0 (socket says 172.21.15.89:46682) 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 -- 172.21.15.89:0/1681095709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 msgr2=0x7f52181954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 -- 172.21.15.89:0/1681095709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218101390 msgr2=0x7f521819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218101390 0x7f521819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 -- 172.21.15.89:0/1681095709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52080095f0 con 0x7f52181009f0 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521dc82700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:25:04.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521d481700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5208009fd0 tx=0x7f520800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520801d070 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521819d580 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5208005020 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208004a50 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.751+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521819d950 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.752+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5218106d80 con 0x7f52181009f0 2023-08-25T02:25:04.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.753+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5208005400 con 0x7f52181009f0 2023-08-25T02:25:04.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.753+0000 7f520effd700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 0x7f52040a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:04.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.753+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f52080cbd20 con 0x7f52181009f0 2023-08-25T02:25:04.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.753+0000 7f521dc82700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 0x7f52040a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:04.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.754+0000 7f521dc82700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 0x7f52040a2460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5214005950 tx=0x7f5214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:04.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.757+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5208095c90 con 0x7f52181009f0 2023-08-25T02:25:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.954+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521810b1e0 con 0x7f52181009f0 2023-08-25T02:25:04.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.955+0000 7f520effd700 1 -- 172.21.15.89:0/1681095709 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22120 (secure 0 0 0) 0x7f5208005190 con 0x7f52181009f0 2023-08-25T02:25:04.957 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:04.958 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":79,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:04.023668+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 msgr2=0x7f52040a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 0x7f52040a2460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5214005950 tx=0x7f5214009260 comp rx=0 tx=0).stop 2023-08-25T02:25:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 msgr2=0x7f52181959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5208009fd0 tx=0x7f520800f690 comp rx=0 tx=0).stop 2023-08-25T02:25:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 shutdown_connections 2023-08-25T02:25:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f520409ffa0 0x7f52040a2460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218101390 0x7f521819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52180ff800 0x7f52181954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 --2- 172.21.15.89:0/1681095709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52181009f0 0x7f52181959f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 >> 172.21.15.89:0/1681095709 conn(0x7f52180fad50 msgr2=0x7f5218103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 shutdown_connections 2023-08-25T02:25:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:04.958+0000 7f521fee6700 1 -- 172.21.15.89:0/1681095709 wait complete. 2023-08-25T02:25:04.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: pgmap v247: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:04.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-08-25T02:25:04.970 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: osdmap e78: 12 total, 12 up, 12 in 2023-08-25T02:25:04.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-08-25T02:25:04.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-08-25T02:25:04.971 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-08-25T02:25:04.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:04 smithi089 ceph-mon[98732]: osdmap e79: 12 total, 12 up, 12 in 2023-08-25T02:25:04.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: pgmap v247: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:25:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1148929684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-08-25T02:25:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: osdmap e78: 12 total, 12 up, 12 in 2023-08-25T02:25:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-08-25T02:25:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-08-25T02:25:04.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-08-25T02:25:04.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:04 smithi122 ceph-mon[100457]: osdmap e79: 12 total, 12 up, 12 in 2023-08-25T02:25:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- 172.21.15.89:0/2065870167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 msgr2=0x7f7464101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 --2- 172.21.15.89:0/2065870167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464101f70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f74500099c0 tx=0x7f7450009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- 172.21.15.89:0/2065870167 shutdown_connections 2023-08-25T02:25:05.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 --2- 172.21.15.89:0/2065870167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 0x7f7464100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 --2- 172.21.15.89:0/2065870167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 0x7f7464108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 --2- 172.21.15.89:0/2065870167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464101f70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- 172.21.15.89:0/2065870167 >> 172.21.15.89:0/2065870167 conn(0x7f74640fbe70 msgr2=0x7f74640fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:05.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- 172.21.15.89:0/2065870167 shutdown_connections 2023-08-25T02:25:05.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- 172.21.15.89:0/2065870167 wait complete. 2023-08-25T02:25:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 Processor -- start 2023-08-25T02:25:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.102+0000 7f746ad12700 1 -- start start 2023-08-25T02:25:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 0x7f7464196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 0x7f746419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419e190 con 0x7f7464101b10 2023-08-25T02:25:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419e2d0 con 0x7f7464100920 2023-08-25T02:25:05.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419e460 con 0x7f74641024b0 2023-08-25T02:25:05.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46696/0 (socket says 172.21.15.89:46696) 2023-08-25T02:25:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 -- 172.21.15.89:0/1846631594 learned_addr learned my addr 172.21.15.89:0/1846631594 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7468aae700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 0x7f7464196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f74692af700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 0x7f746419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 -- 172.21.15.89:0/1846631594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 msgr2=0x7f746419dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 0x7f746419dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 -- 172.21.15.89:0/1846631594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 msgr2=0x7f7464196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 0x7f7464196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 -- 172.21.15.89:0/1846631594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74500095f0 con 0x7f7464101b10 2023-08-25T02:25:05.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7463fff700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fd0 tx=0x7f745000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:05.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745001d070 con 0x7f7464101b10 2023-08-25T02:25:05.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7450005030 con 0x7f7464101b10 2023-08-25T02:25:05.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746419e6e0 con 0x7f7464101b10 2023-08-25T02:25:05.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.103+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450004b30 con 0x7f7464101b10 2023-08-25T02:25:05.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.104+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746419eb60 con 0x7f7464101b10 2023-08-25T02:25:05.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.104+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7464107ec0 con 0x7f7464101b10 2023-08-25T02:25:05.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.105+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f7450004cd0 con 0x7f7464101b10 2023-08-25T02:25:05.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.106+0000 7f7461ffb700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 0x7f74540a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.106+0000 7f7468aae700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 0x7f74540a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.106+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f74500cbf40 con 0x7f7464101b10 2023-08-25T02:25:05.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.106+0000 7f7468aae700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 0x7f74540a2560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7464101970 tx=0x7f7458007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:05.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.110+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7450095e10 con 0x7f7464101b10 2023-08-25T02:25:05.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.330+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f746410c130 con 0x7f7464101b10 2023-08-25T02:25:05.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.331+0000 7f7461ffb700 1 -- 172.21.15.89:0/1846631594 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f7450099430 con 0x7f7464101b10 2023-08-25T02:25:05.332 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:05.333 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:05.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 msgr2=0x7f74540a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 0x7f74540a2560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7464101970 tx=0x7f7458007400 comp rx=0 tx=0).stop 2023-08-25T02:25:05.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 msgr2=0x7f7464196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fd0 tx=0x7f745000f690 comp rx=0 tx=0).stop 2023-08-25T02:25:05.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 shutdown_connections 2023-08-25T02:25:05.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f74540a00a0 0x7f74540a2560 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7464100920 0x7f7464196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74641024b0 0x7f746419dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 --2- 172.21.15.89:0/1846631594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7464101b10 0x7f7464196b50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 >> 172.21.15.89:0/1846631594 conn(0x7f74640fbe70 msgr2=0x7f7464105000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:05.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.335+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 shutdown_connections 2023-08-25T02:25:05.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.336+0000 7f746ad12700 1 -- 172.21.15.89:0/1846631594 wait complete. 2023-08-25T02:25:05.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-08-25T02:25:05.468 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: 2023-08-25T02:25:05.148+0000 7f31f3a6e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-08-25T02:25:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 -- 172.21.15.89:0/2801618579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58101390 msgr2=0x7fad58103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 --2- 172.21.15.89:0/2801618579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58101390 0x7fad58103780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fad48009990 tx=0x7fad48009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 -- 172.21.15.89:0/2801618579 shutdown_connections 2023-08-25T02:25:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 --2- 172.21.15.89:0/2801618579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad581065f0 0x7fad58108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 --2- 172.21.15.89:0/2801618579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58101390 0x7fad58103780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 --2- 172.21.15.89:0/2801618579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad58103cc0 0x7fad581060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 -- 172.21.15.89:0/2801618579 >> 172.21.15.89:0/2801618579 conn(0x7fad580fad80 msgr2=0x7fad580fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.478+0000 7fad5d33f700 1 -- 172.21.15.89:0/2801618579 shutdown_connections 2023-08-25T02:25:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 -- 172.21.15.89:0/2801618579 wait complete. 2023-08-25T02:25:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 Processor -- start 2023-08-25T02:25:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 -- start start 2023-08-25T02:25:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad58101390 0x7fad58196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad581065f0 0x7fad5819dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5819e1b0 con 0x7fad58103cc0 2023-08-25T02:25:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5819e2f0 con 0x7fad581065f0 2023-08-25T02:25:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.479+0000 7fad5d33f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5819e480 con 0x7fad58101390 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46708/0 (socket says 172.21.15.89:46708) 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 -- 172.21.15.89:0/3189803964 learned_addr learned my addr 172.21.15.89:0/3189803964 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad56ffd700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad58101390 0x7fad58196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad577fe700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad581065f0 0x7fad5819dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 -- 172.21.15.89:0/3189803964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad58101390 msgr2=0x7fad58196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad58101390 0x7fad58196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 -- 172.21.15.89:0/3189803964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad581065f0 msgr2=0x7fad5819dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad581065f0 0x7fad5819dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 -- 172.21.15.89:0/3189803964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480095f0 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad567fc700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00e9a0 tx=0x7fad4c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c009790 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad4c010430 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c018680 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5819e760 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.480+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5819ec20 con 0x7fad58103cc0 2023-08-25T02:25:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.481+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad580fc900 con 0x7fad58103cc0 2023-08-25T02:25:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.483+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fad4c0105a0 con 0x7fad58103cc0 2023-08-25T02:25:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.483+0000 7fad3ffff700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 0x7fad400a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.484+0000 7fad56ffd700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 0x7fad400a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.484+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fad4c014070 con 0x7fad58103cc0 2023-08-25T02:25:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.484+0000 7fad56ffd700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 0x7fad400a2410 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad48006010 tx=0x7fad48005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.488+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fad4c051e00 con 0x7fad58103cc0 2023-08-25T02:25:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:05.735+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fad5810c140 con 0x7fad58103cc0 2023-08-25T02:25:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-08-25T02:25:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-08-25T02:25:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-08-25T02:25:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1681095709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:25:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: osdmap e80: 12 total, 12 up, 12 in 2023-08-25T02:25:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:05 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1846631594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-08-25T02:25:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-08-25T02:25:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-08-25T02:25:05.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1681095709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:05.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:25:05.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:05.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: osdmap e80: 12 total, 12 up, 12 in 2023-08-25T02:25:05.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:05 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1846631594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:06.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/28562118' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-08-25T02:25:06.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-08-25T02:25:06.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-08-25T02:25:06.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1681095709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:06.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:25:06.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:06.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: osdmap e80: 12 total, 12 up, 12 in 2023-08-25T02:25:06.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:05 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1846631594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.608+0000 7fad3ffff700 1 -- 172.21.15.89:0/3189803964 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fad4c095180 con 0x7fad58103cc0 2023-08-25T02:25:06.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 msgr2=0x7fad400a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:06.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 0x7fad400a2410 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad48006010 tx=0x7fad48005a90 comp rx=0 tx=0).stop 2023-08-25T02:25:06.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 msgr2=0x7fad58196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:06.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00e9a0 tx=0x7fad4c00ecb0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 shutdown_connections 2023-08-25T02:25:06.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fad4009ff50 0x7fad400a2410 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad581065f0 0x7fad5819dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad58101390 0x7fad58196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 --2- 172.21.15.89:0/3189803964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad58103cc0 0x7fad58196b70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 >> 172.21.15.89:0/3189803964 conn(0x7fad580fad80 msgr2=0x7fad58102000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 shutdown_connections 2023-08-25T02:25:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.611+0000 7fad5d33f700 1 -- 172.21.15.89:0/3189803964 wait complete. 2023-08-25T02:25:06.625 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True} 2023-08-25T02:25:06.626 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-08-25T02:25:06.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- 172.21.15.89:0/1291625315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4101b20 msgr2=0x7f76e4101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:06.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 --2- 172.21.15.89:0/1291625315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4101b20 0x7f76e4101f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76e00099c0 tx=0x7f76e0009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- 172.21.15.89:0/1291625315 shutdown_connections 2023-08-25T02:25:06.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 --2- 172.21.15.89:0/1291625315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4100930 0x7f76e4100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 --2- 172.21.15.89:0/1291625315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 0x7f76e4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 --2- 172.21.15.89:0/1291625315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4101b20 0x7f76e4101f80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- 172.21.15.89:0/1291625315 >> 172.21.15.89:0/1291625315 conn(0x7f76e40fbea0 msgr2=0x7f76e40fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- 172.21.15.89:0/1291625315 shutdown_connections 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- 172.21.15.89:0/1291625315 wait complete. 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 Processor -- start 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.753+0000 7f76ebd14700 1 -- start start 2023-08-25T02:25:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:06.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4101b20 0x7f76e4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:06.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 0x7f76e419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:06.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e419e0b0 con 0x7f76e4100930 2023-08-25T02:25:06.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e419e1f0 con 0x7f76e4101b20 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e419e380 con 0x7f76e41024c0 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46720/0 (socket says 172.21.15.89:46720) 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 -- 172.21.15.89:0/4096920854 learned_addr learned my addr 172.21.15.89:0/4096920854 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ea2b1700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 0x7f76e419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:06.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e92af700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4101b20 0x7f76e4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 -- 172.21.15.89:0/4096920854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 msgr2=0x7f76e419dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 0x7f76e419dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 -- 172.21.15.89:0/4096920854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4101b20 msgr2=0x7f76e4196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4101b20 0x7f76e4196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 -- 172.21.15.89:0/4096920854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00095f0 con 0x7f76e4100930 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76e9ab0700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d400c990 tx=0x7f76d400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4007b00 con 0x7f76e4100930 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76d4007c60 con 0x7f76e4100930 2023-08-25T02:25:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e419e660 con 0x7f76e4100930 2023-08-25T02:25:06.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4018660 con 0x7f76e4100930 2023-08-25T02:25:06.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.754+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e419eb70 con 0x7f76e4100930 2023-08-25T02:25:06.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.755+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e4107e50 con 0x7f76e4100930 2023-08-25T02:25:06.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.757+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f76d400f450 con 0x7f76e4100930 2023-08-25T02:25:06.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.757+0000 7f76daffd700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 0x7f76d00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:06.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.758+0000 7f76e92af700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 0x7f76d00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:06.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.758+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f76d40cb100 con 0x7f76e4100930 2023-08-25T02:25:06.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.758+0000 7f76e92af700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 0x7f76d00a2410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76e0000c00 tx=0x7f76e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:06.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.761+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f76d4095070 con 0x7f76e4100930 2023-08-25T02:25:06.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: pgmap v251: 97 pgs: 14 creating+peering, 20 active+clean, 63 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:06.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:06.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-08-25T02:25:06.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:06 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:25:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: pgmap v251: 97 pgs: 14 creating+peering, 20 active+clean, 63 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-08-25T02:25:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-08-25T02:25:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:06.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-08-25T02:25:06.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:06 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:25:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:06.997+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f76e410c2d0 con 0x7f76e4100930 2023-08-25T02:25:07.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: pgmap v251: 97 pgs: 14 creating+peering, 20 active+clean, 63 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2023-08-25T02:25:07.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-08-25T02:25:07.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:25:07.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:25:07.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:25:07.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-08-25T02:25:07.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:06 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:25:07.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.617+0000 7f76daffd700 1 -- 172.21.15.89:0/4096920854 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) v1 ==== 105+0+0 (secure 0 0 0) 0x7f76d4098690 con 0x7f76e4100930 2023-08-25T02:25:07.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 msgr2=0x7f76d00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:07.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 0x7f76d00a2410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76e0000c00 tx=0x7f76e0005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 msgr2=0x7f76e4196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:07.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d400c990 tx=0x7f76d400cca0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 shutdown_connections 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e41024c0 0x7f76e419dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76e4100930 0x7f76e4196530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f76d009ff50 0x7f76d00a2410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 --2- 172.21.15.89:0/4096920854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76e4101b20 0x7f76e4196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 >> 172.21.15.89:0/4096920854 conn(0x7f76e40fbea0 msgr2=0x7f76e4104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 shutdown_connections 2023-08-25T02:25:07.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.619+0000 7f76ebd14700 1 -- 172.21.15.89:0/4096920854 wait complete. 2023-08-25T02:25:07.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-08-25T02:25:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 -- 172.21.15.89:0/332551787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 msgr2=0x7fd3080ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/332551787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3080ffc30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0099c0 tx=0x7fd2fc009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 -- 172.21.15.89:0/332551787 shutdown_connections 2023-08-25T02:25:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/332551787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd308107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/332551787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3080ffc30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/332551787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 -- 172.21.15.89:0/332551787 >> 172.21.15.89:0/332551787 conn(0x7fd3080fad80 msgr2=0x7fd3080fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:07.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 -- 172.21.15.89:0/332551787 shutdown_connections 2023-08-25T02:25:07.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.760+0000 7fd30ccb7700 1 -- 172.21.15.89:0/332551787 wait complete. 2023-08-25T02:25:07.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 Processor -- start 2023-08-25T02:25:07.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 -- start start 2023-08-25T02:25:07.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd30819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30819e140 con 0x7fd3080ff810 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30819e280 con 0x7fd3081013a0 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30819e410 con 0x7fd308100a00 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:07.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46734/0 (socket says 172.21.15.89:46734) 2023-08-25T02:25:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 -- 172.21.15.89:0/2925347686 learned_addr learned my addr 172.21.15.89:0/2925347686 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd305d9b700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd306d9d700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd30819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 -- 172.21.15.89:0/2925347686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 msgr2=0x7fd308196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:07.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 -- 172.21.15.89:0/2925347686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 msgr2=0x7fd30819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:07.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd30819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:07.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 -- 172.21.15.89:0/2925347686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0095f0 con 0x7fd3080ff810 2023-08-25T02:25:07.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd306d9d700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd30819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd305d9b700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30659c700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009f50 tx=0x7fd2fc0057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc01d070 con 0x7fd3080ff810 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.761+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30819e690 con 0x7fd3080ff810 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.762+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2fc004110 con 0x7fd3080ff810 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.762+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc022dd0 con 0x7fd3080ff810 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.762+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30819eb90 con 0x7fd3080ff810 2023-08-25T02:25:07.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.762+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd308107940 con 0x7fd3080ff810 2023-08-25T02:25:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.763+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd2fc022500 con 0x7fd3080ff810 2023-08-25T02:25:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.764+0000 7fd2f77fe700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 0x7fd2f00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.764+0000 7fd305d9b700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 0x7fd2f00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.764+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd2fc0cc510 con 0x7fd3080ff810 2023-08-25T02:25:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.764+0000 7fd305d9b700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 0x7fd2f00a2410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005fd0 tx=0x7fd2f8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:07.768+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd2fc096480 con 0x7fd3080ff810 2023-08-25T02:25:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:07 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-08-25T02:25:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:07 smithi121 ceph-mon[100099]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:07 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-08-25T02:25:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:07 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-08-25T02:25:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:07 smithi122 ceph-mon[100457]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:07 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-08-25T02:25:08.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.004+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd30810b020 con 0x7fd3080ff810 2023-08-25T02:25:08.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:07 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3189803964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-08-25T02:25:08.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:07 smithi089 ceph-mon[98732]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:08.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:07 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-08-25T02:25:08.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.624+0000 7fd2f77fe700 1 -- 172.21.15.89:0/2925347686 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd2fc0270c0 con 0x7fd3080ff810 2023-08-25T02:25:08.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.626+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 msgr2=0x7fd2f00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:08.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.626+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 0x7fd2f00a2410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005fd0 tx=0x7fd2f8005e20 comp rx=0 tx=0).stop 2023-08-25T02:25:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.626+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 msgr2=0x7fd3081965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.626+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009f50 tx=0x7fd2fc0057d0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 shutdown_connections 2023-08-25T02:25:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd2f009ff50 0x7fd2f00a2410 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3081013a0 0x7fd30819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3080ff810 0x7fd3081965c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 --2- 172.21.15.89:0/2925347686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd308100a00 0x7fd308196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 >> 172.21.15.89:0/2925347686 conn(0x7fd3080fad80 msgr2=0x7fd3080fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 shutdown_connections 2023-08-25T02:25:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.627+0000 7fd30ccb7700 1 -- 172.21.15.89:0/2925347686 wait complete. 2023-08-25T02:25:08.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-08-25T02:25:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- 172.21.15.89:0/653722072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 msgr2=0x7eff70100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 --2- 172.21.15.89:0/653722072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70100d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7eff640099c0 tx=0x7eff64009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- 172.21.15.89:0/653722072 shutdown_connections 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 --2- 172.21.15.89:0/653722072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 0x7eff70108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 --2- 172.21.15.89:0/653722072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70100d30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 --2- 172.21.15.89:0/653722072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 0x7eff70101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- 172.21.15.89:0/653722072 >> 172.21.15.89:0/653722072 conn(0x7eff700fbe60 msgr2=0x7eff700fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- 172.21.15.89:0/653722072 shutdown_connections 2023-08-25T02:25:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- 172.21.15.89:0/653722072 wait complete. 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 Processor -- start 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.771+0000 7eff76e28700 1 -- start start 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 0x7eff70196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 0x7eff7019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7019e190 con 0x7eff70100910 2023-08-25T02:25:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7019e2d0 con 0x7eff701024a0 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7019e460 con 0x7eff70101b00 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46748/0 (socket says 172.21.15.89:46748) 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 -- 172.21.15.89:0/2725725203 learned_addr learned my addr 172.21.15.89:0/2725725203 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff753c5700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 0x7eff7019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff6ffff700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 0x7eff70196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 -- 172.21.15.89:0/2725725203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 msgr2=0x7eff70196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 0x7eff70196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 -- 172.21.15.89:0/2725725203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 msgr2=0x7eff7019dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 0x7eff7019dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 -- 172.21.15.89:0/2725725203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff640095f0 con 0x7eff70100910 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff74bc4700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7eff64009990 tx=0x7eff64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6401d070 con 0x7eff70100910 2023-08-25T02:25:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff64004090 con 0x7eff70100910 2023-08-25T02:25:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7019e6e0 con 0x7eff70100910 2023-08-25T02:25:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.772+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64022d00 con 0x7eff70100910 2023-08-25T02:25:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.773+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7019ebe0 con 0x7eff70100910 2023-08-25T02:25:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.774+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff701078c0 con 0x7eff70100910 2023-08-25T02:25:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.775+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7eff64022470 con 0x7eff70100910 2023-08-25T02:25:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.776+0000 7eff6dffb700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 0x7eff600a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.776+0000 7eff6ffff700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 0x7eff600a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.776+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7eff640cc970 con 0x7eff70100910 2023-08-25T02:25:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.776+0000 7eff6ffff700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 0x7eff600a2490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff5c005950 tx=0x7eff5c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:08.779+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7eff64096840 con 0x7eff70100910 2023-08-25T02:25:08.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: pgmap v252: 97 pgs: 14 creating+peering, 66 active+clean, 17 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 3 op/s 2023-08-25T02:25:08.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-08-25T02:25:08.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-08-25T02:25:08.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-08-25T02:25:08.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:08.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-08-25T02:25:08.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-08-25T02:25:08.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:08 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-08-25T02:25:08.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: pgmap v252: 97 pgs: 14 creating+peering, 66 active+clean, 17 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 3 op/s 2023-08-25T02:25:08.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-08-25T02:25:08.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-08-25T02:25:08.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-08-25T02:25:08.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:08.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-08-25T02:25:08.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-08-25T02:25:08.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:08 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-08-25T02:25:09.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.016+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7eff7010c120 con 0x7eff70100910 2023-08-25T02:25:09.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: pgmap v252: 97 pgs: 14 creating+peering, 66 active+clean, 17 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 3 op/s 2023-08-25T02:25:09.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4096920854' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-08-25T02:25:09.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-08-25T02:25:09.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.634+0000 7eff6dffb700 1 -- 172.21.15.89:0/2725725203 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7eff64022e60 con 0x7eff70100910 2023-08-25T02:25:09.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 msgr2=0x7eff600a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:09.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 0x7eff600a2490 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff5c005950 tx=0x7eff5c0058e0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 msgr2=0x7eff70196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:09.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7eff64009990 tx=0x7eff64004730 comp rx=0 tx=0).stop 2023-08-25T02:25:09.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 shutdown_connections 2023-08-25T02:25:09.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7eff6009ffd0 0x7eff600a2490 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff701024a0 0x7eff7019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100910 0x7eff70196610 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 --2- 172.21.15.89:0/2725725203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff70101b00 0x7eff70196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.637+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 >> 172.21.15.89:0/2725725203 conn(0x7eff700fbe60 msgr2=0x7eff700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:09.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.638+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 shutdown_connections 2023-08-25T02:25:09.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.638+0000 7eff76e28700 1 -- 172.21.15.89:0/2725725203 wait complete. 2023-08-25T02:25:09.653 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-08-25T02:25:09.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:25:09.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/3078076481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a980ff810 msgr2=0x7f1a980ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:09.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/3078076481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a980ff810 0x7f1a980ffc30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1a940099c0 tx=0x7f1a94009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/3078076481 shutdown_connections 2023-08-25T02:25:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/3078076481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a981013a0 0x7f1a98107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/3078076481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a980ff810 0x7f1a980ffc30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/3078076481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a98100a00 0x7f1a98100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.777+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/3078076481 >> 172.21.15.89:0/3078076481 conn(0x7f1a980fad80 msgr2=0x7f1a980fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/3078076481 shutdown_connections 2023-08-25T02:25:09.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/3078076481 wait complete. 2023-08-25T02:25:09.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 Processor -- start 2023-08-25T02:25:09.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- start start 2023-08-25T02:25:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a980ff810 0x7f1a981037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 0x7f1a98101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a98102ac0 con 0x7f1a981013a0 2023-08-25T02:25:09.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a98102c30 con 0x7f1a98100a00 2023-08-25T02:25:09.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9f6a0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a981a05b0 con 0x7f1a980ff810 2023-08-25T02:25:09.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9dc3d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:09.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9dc3d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46766/0 (socket says 172.21.15.89:46766) 2023-08-25T02:25:09.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.778+0000 7f1a9dc3d700 1 -- 172.21.15.89:0/4032376897 learned_addr learned my addr 172.21.15.89:0/4032376897 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:09.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9cc3b700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 0x7f1a98101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:09.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 -- 172.21.15.89:0/4032376897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a980ff810 msgr2=0x7f1a981037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:09.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9d43c700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a980ff810 0x7f1a981037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:09.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a980ff810 0x7f1a981037a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 -- 172.21.15.89:0/4032376897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 msgr2=0x7f1a98101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:09.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 0x7f1a98101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:09.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 -- 172.21.15.89:0/4032376897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940095f0 con 0x7f1a981013a0 2023-08-25T02:25:09.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9cc3b700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 0x7f1a98101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9dc3d700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1a90007d70 tx=0x7f1a9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9000ad40 con 0x7f1a981013a0 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a900128e0 con 0x7f1a981013a0 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a981a0890 con 0x7f1a981013a0 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90011780 con 0x7f1a981013a0 2023-08-25T02:25:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.779+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a981a0d50 con 0x7f1a981013a0 2023-08-25T02:25:09.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.780+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9810b020 con 0x7f1a981013a0 2023-08-25T02:25:09.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.781+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f1a90012a50 con 0x7f1a981013a0 2023-08-25T02:25:09.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.782+0000 7f1a8e7fc700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 0x7f1a840a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:09.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.782+0000 7f1a9d43c700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 0x7f1a840a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:09.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.782+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1a900cb860 con 0x7f1a981013a0 2023-08-25T02:25:09.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.782+0000 7f1a9d43c700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 0x7f1a840a2460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a940098e0 tx=0x7f1a9400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:09.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:09.785+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1a90095740 con 0x7f1a981013a0 2023-08-25T02:25:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:25:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-08-25T02:25:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-08-25T02:25:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-08-25T02:25:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:25:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-08-25T02:25:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-08-25T02:25:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-08-25T02:25:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:25:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-08-25T02:25:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-08-25T02:25:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-08-25T02:25:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-08-25T02:25:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.020+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a9810b210 con 0x7f1a981013a0 2023-08-25T02:25:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.021+0000 7f1a8e7fc700 1 -- 172.21.15.89:0/4032376897 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7f1a90098d60 con 0x7f1a981013a0 2023-08-25T02:25:10.022 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:10.022 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:25:10.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 msgr2=0x7f1a840a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 0x7f1a840a2460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a940098e0 tx=0x7f1a9400b540 comp rx=0 tx=0).stop 2023-08-25T02:25:10.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 msgr2=0x7f1a98102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1a90007d70 tx=0x7f1a9000a340 comp rx=0 tx=0).stop 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 shutdown_connections 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a981013a0 0x7f1a98102320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a980ff810 0x7f1a981037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f1a8409ffa0 0x7f1a840a2460 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 --2- 172.21.15.89:0/4032376897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a98100a00 0x7f1a98101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 >> 172.21.15.89:0/4032376897 conn(0x7f1a980fad80 msgr2=0x7f1a98068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 shutdown_connections 2023-08-25T02:25:10.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.023+0000 7f1a9f6a0700 1 -- 172.21.15.89:0/4032376897 wait complete. 2023-08-25T02:25:10.028 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:25:10.035 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:25:10.036 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2925347686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-08-25T02:25:10.036 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-08-25T02:25:10.036 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-08-25T02:25:10.036 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:25:10.036 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-08-25T02:25:10.037 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-08-25T02:25:10.037 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-08-25T02:25:10.037 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-08-25T02:25:10.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:10.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- 172.21.15.89:0/3246411537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 msgr2=0x7f42e4100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- 172.21.15.89:0/3246411537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e4100ea0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42d00099c0 tx=0x7f42d0009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- 172.21.15.89:0/3246411537 shutdown_connections 2023-08-25T02:25:10.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- 172.21.15.89:0/3246411537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e4107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- 172.21.15.89:0/3246411537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 0x7f42e40ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- 172.21.15.89:0/3246411537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e4100ea0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- 172.21.15.89:0/3246411537 >> 172.21.15.89:0/3246411537 conn(0x7f42e40fada0 msgr2=0x7f42e40fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- 172.21.15.89:0/3246411537 shutdown_connections 2023-08-25T02:25:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- 172.21.15.89:0/3246411537 wait complete. 2023-08-25T02:25:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 Processor -- start 2023-08-25T02:25:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- start start 2023-08-25T02:25:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 0x7f42e4195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e419cfe0 con 0x7f42e4100a40 2023-08-25T02:25:10.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e419d120 con 0x7f42e41013e0 2023-08-25T02:25:10.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.160+0000 7f42eab43700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e419d2b0 con 0x7f42e40ff850 2023-08-25T02:25:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46778/0 (socket says 172.21.15.89:46778) 2023-08-25T02:25:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 -- 172.21.15.89:0/221879686 learned_addr learned my addr 172.21.15.89:0/221879686 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e88df700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 0x7f42e4195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e90e0700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 -- 172.21.15.89:0/221879686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 msgr2=0x7f42e4195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 0x7f42e4195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 -- 172.21.15.89:0/221879686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 msgr2=0x7f42e419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e419ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 -- 172.21.15.89:0/221879686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00095f0 con 0x7f42e4100a40 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e90e0700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e3fff700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009990 tx=0x7f42d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d001d070 con 0x7f42e4100a40 2023-08-25T02:25:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e419d530 con 0x7f42e4100a40 2023-08-25T02:25:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42d0004090 con 0x7f42e4100a40 2023-08-25T02:25:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0004a00 con 0x7f42e4100a40 2023-08-25T02:25:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.161+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e419d9e0 con 0x7f42e4100a40 2023-08-25T02:25:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.162+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e410b060 con 0x7f42e4100a40 2023-08-25T02:25:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.163+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f42d0003bd0 con 0x7f42e4100a40 2023-08-25T02:25:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.164+0000 7f42e1ffb700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 0x7f42d40a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.164+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f42d00cc3b0 con 0x7f42e4100a40 2023-08-25T02:25:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.164+0000 7f42e88df700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 0x7f42d40a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.164+0000 7f42e88df700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 0x7f42d40a2460 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f42e41008a0 tx=0x7f42d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.168+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f42d0096320 con 0x7f42e4100a40 2023-08-25T02:25:10.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.360+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e410b270 con 0x7f42e4100a40 2023-08-25T02:25:10.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.361+0000 7f42e1ffb700 1 -- 172.21.15.89:0/221879686 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f42d0099940 con 0x7f42e4100a40 2023-08-25T02:25:10.363 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:10.363 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:10.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.363+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 msgr2=0x7f42d40a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.363+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 0x7f42d40a2460 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f42e41008a0 tx=0x7f42d800b410 comp rx=0 tx=0).stop 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.363+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 msgr2=0x7f42e41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.363+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009990 tx=0x7f42d0004730 comp rx=0 tx=0).stop 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 shutdown_connections 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f42d409ffa0 0x7f42d40a2460 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e41013e0 0x7f42e419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e40ff850 0x7f42e4195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 --2- 172.21.15.89:0/221879686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e4100a40 0x7f42e41959a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 >> 172.21.15.89:0/221879686 conn(0x7f42e40fada0 msgr2=0x7f42e40fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 shutdown_connections 2023-08-25T02:25:10.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.364+0000 7f42eab43700 1 -- 172.21.15.89:0/221879686 wait complete. 2023-08-25T02:25:10.377 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-08-25T02:25:10.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T02:25:10.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 -- 172.21.15.89:0/3264323204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 msgr2=0x7fa4e40ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/3264323204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e40ff3b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80099c0 tx=0x7fa4d8009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 -- 172.21.15.89:0/3264323204 shutdown_connections 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/3264323204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 0x7fa4e41073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/3264323204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e40ff3b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/3264323204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 0x7fa4e41005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 -- 172.21.15.89:0/3264323204 >> 172.21.15.89:0/3264323204 conn(0x7fa4e40fa4e0 msgr2=0x7fa4e40fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 -- 172.21.15.89:0/3264323204 shutdown_connections 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 -- 172.21.15.89:0/3264323204 wait complete. 2023-08-25T02:25:10.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.498+0000 7fa4e97cc700 1 Processor -- start 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- start start 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 0x7fa4e4101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 0x7fa4e4101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e4102240 con 0x7fa4e40fef90 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e41023b0 con 0x7fa4e4100180 2023-08-25T02:25:10.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e41a06b0 con 0x7fa4e4100b20 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46790/0 (socket says 172.21.15.89:46790) 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 -- 172.21.15.89:0/910559575 learned_addr learned my addr 172.21.15.89:0/910559575 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e37fe700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 0x7fa4e4101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 -- 172.21.15.89:0/910559575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 msgr2=0x7fa4e4101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 0x7fa4e4101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e27fc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 0x7fa4e4101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 -- 172.21.15.89:0/910559575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 msgr2=0x7fa4e4101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 0x7fa4e4101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 -- 172.21.15.89:0/910559575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80095f0 con 0x7fa4e40fef90 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e2ffd700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80036f0 tx=0x7fa4d80038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d801d070 con 0x7fa4e40fef90 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4d8022470 con 0x7fa4e40fef90 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e41a0930 con 0x7fa4e40fef90 2023-08-25T02:25:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800f6c0 con 0x7fa4e40fef90 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.499+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e41a0d50 con 0x7fa4e40fef90 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.500+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e410b080 con 0x7fa4e40fef90 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.501+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fa4d80225e0 con 0x7fa4e40fef90 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.502+0000 7fa4cbfff700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 0x7fa4cc0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.502+0000 7fa4e27fc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 0x7fa4cc0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.502+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa4d80ccdd0 con 0x7fa4e40fef90 2023-08-25T02:25:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.502+0000 7fa4e27fc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 0x7fa4cc0a2460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4005950 tx=0x7fa4d4005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.505+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa4d8096d40 con 0x7fa4e40fef90 2023-08-25T02:25:10.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.693+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4e4108450 con 0x7fa4cc09ffa0 2023-08-25T02:25:10.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.704+0000 7fa4cbfff700 1 -- 172.21.15.89:0/910559575 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335584 (secure 0 0 0) 0x7fa4e4108450 con 0x7fa4cc09ffa0 2023-08-25T02:25:10.709 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:10.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 msgr2=0x7fa4cc0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 0x7fa4cc0a2460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4005950 tx=0x7fa4d4005cf0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 msgr2=0x7fa4e4102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80036f0 tx=0x7fa4d80038c0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 shutdown_connections 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e4100b20 0x7fa4e4101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4e40fef90 0x7fa4e4102f20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa4cc09ffa0 0x7fa4cc0a2460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 --2- 172.21.15.89:0/910559575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4e4100180 0x7fa4e4101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 >> 172.21.15.89:0/910559575 conn(0x7fa4e40fa4e0 msgr2=0x7fa4e4107d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 shutdown_connections 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.708+0000 7fa4e97cc700 1 -- 172.21.15.89:0/910559575 wait complete. 2023-08-25T02:25:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-08-25T02:25:10.722 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2023-08-25T02:25:09.296350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594004,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":244,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336056,"kb_used_data":11808,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605768,"statfs":{"total":1151940427776,"available":1151596306432,"internally_reserved":0,"allocated":12091392,"data_stored":6661363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3636,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001210"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.227164+0000","last_change":"2023-08-25T02:25:04.226503+0000","last_active":"2023-08-25T02:25:04.227164+0000","last_peered":"2023-08-25T02:25:04.227164+0000","last_clean":"2023-08-25T02:25:04.227164+0000","last_became_active":"2023-08-25T02:25:04.226184+0000","last_became_peered":"2023-08-25T02:25:04.226184+0000","last_unstale":"2023-08-25T02:25:04.227164+0000","last_undegraded":"2023-08-25T02:25:04.227164+0000","last_fullsized":"2023-08-25T02:25:04.227164+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:12:59.270865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938361+0000","last_change":"2023-08-25T02:25:04.236479+0000","last_active":"2023-08-25T02:25:05.938361+0000","last_peered":"2023-08-25T02:25:05.938361+0000","last_clean":"2023-08-25T02:25:05.938361+0000","last_became_active":"2023-08-25T02:25:04.235376+0000","last_became_peered":"2023-08-25T02:25:04.235376+0000","last_unstale":"2023-08-25T02:25:05.938361+0000","last_undegraded":"2023-08-25T02:25:05.938361+0000","last_fullsized":"2023-08-25T02:25:05.938361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:47.418998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938362+0000","last_change":"2023-08-25T02:25:04.235840+0000","last_active":"2023-08-25T02:25:05.938362+0000","last_peered":"2023-08-25T02:25:05.938362+0000","last_clean":"2023-08-25T02:25:05.938362+0000","last_became_active":"2023-08-25T02:25:04.235279+0000","last_became_peered":"2023-08-25T02:25:04.235279+0000","last_unstale":"2023-08-25T02:25:05.938362+0000","last_undegraded":"2023-08-25T02:25:05.938362+0000","last_fullsized":"2023-08-25T02:25:05.938362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:23.103019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474317+0000","last_change":"2023-08-25T02:25:04.237793+0000","last_active":"2023-08-25T02:25:06.474317+0000","last_peered":"2023-08-25T02:25:06.474317+0000","last_clean":"2023-08-25T02:25:06.474317+0000","last_became_active":"2023-08-25T02:25:04.237493+0000","last_became_peered":"2023-08-25T02:25:04.237493+0000","last_unstale":"2023-08-25T02:25:06.474317+0000","last_undegraded":"2023-08-25T02:25:06.474317+0000","last_fullsized":"2023-08-25T02:25:06.474317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:36:34.299231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657466+0000","last_change":"2023-08-25T02:25:04.221053+0000","last_active":"2023-08-25T02:25:06.657466+0000","last_peered":"2023-08-25T02:25:06.657466+0000","last_clean":"2023-08-25T02:25:06.657466+0000","last_became_active":"2023-08-25T02:25:04.220710+0000","last_became_peered":"2023-08-25T02:25:04.220710+0000","last_unstale":"2023-08-25T02:25:06.657466+0000","last_undegraded":"2023-08-25T02:25:06.657466+0000","last_fullsized":"2023-08-25T02:25:06.657466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:25:14.107245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214409+0000","last_change":"2023-08-25T02:25:04.234380+0000","last_active":"2023-08-25T02:25:05.214409+0000","last_peered":"2023-08-25T02:25:05.214409+0000","last_clean":"2023-08-25T02:25:05.214409+0000","last_became_active":"2023-08-25T02:25:04.233376+0000","last_became_peered":"2023-08-25T02:25:04.233376+0000","last_unstale":"2023-08-25T02:25:05.214409+0000","last_undegraded":"2023-08-25T02:25:05.214409+0000","last_fullsized":"2023-08-25T02:25:05.214409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:18:49.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473517+0000","last_change":"2023-08-25T02:25:04.228398+0000","last_active":"2023-08-25T02:25:06.473517+0000","last_peered":"2023-08-25T02:25:06.473517+0000","last_clean":"2023-08-25T02:25:06.473517+0000","last_became_active":"2023-08-25T02:25:04.227310+0000","last_became_peered":"2023-08-25T02:25:04.227310+0000","last_unstale":"2023-08-25T02:25:06.473517+0000","last_undegraded":"2023-08-25T02:25:06.473517+0000","last_fullsized":"2023-08-25T02:25:06.473517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:45:46.502139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213776+0000","last_change":"2023-08-25T02:25:04.235175+0000","last_active":"2023-08-25T02:25:05.213776+0000","last_peered":"2023-08-25T02:25:05.213776+0000","last_clean":"2023-08-25T02:25:05.213776+0000","last_became_active":"2023-08-25T02:25:04.234048+0000","last_became_peered":"2023-08-25T02:25:04.234048+0000","last_unstale":"2023-08-25T02:25:05.213776+0000","last_undegraded":"2023-08-25T02:25:05.213776+0000","last_fullsized":"2023-08-25T02:25:05.213776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:20.347389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438643+0000","last_change":"2023-08-25T02:25:04.226475+0000","last_active":"2023-08-25T02:25:06.438643+0000","last_peered":"2023-08-25T02:25:06.438643+0000","last_clean":"2023-08-25T02:25:06.438643+0000","last_became_active":"2023-08-25T02:25:04.226125+0000","last_became_peered":"2023-08-25T02:25:04.226125+0000","last_unstale":"2023-08-25T02:25:06.438643+0000","last_undegraded":"2023-08-25T02:25:06.438643+0000","last_fullsized":"2023-08-25T02:25:06.438643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:44.888916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214553+0000","last_change":"2023-08-25T02:25:04.234476+0000","last_active":"2023-08-25T02:25:05.214553+0000","last_peered":"2023-08-25T02:25:05.214553+0000","last_clean":"2023-08-25T02:25:05.214553+0000","last_became_active":"2023-08-25T02:25:04.234062+0000","last_became_peered":"2023-08-25T02:25:04.234062+0000","last_unstale":"2023-08-25T02:25:05.214553+0000","last_undegraded":"2023-08-25T02:25:05.214553+0000","last_fullsized":"2023-08-25T02:25:05.214553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:35:10.280467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473970+0000","last_change":"2023-08-25T02:25:04.233218+0000","last_active":"2023-08-25T02:25:06.473970+0000","last_peered":"2023-08-25T02:25:06.473970+0000","last_clean":"2023-08-25T02:25:06.473970+0000","last_became_active":"2023-08-25T02:25:04.232933+0000","last_became_peered":"2023-08-25T02:25:04.232933+0000","last_unstale":"2023-08-25T02:25:06.473970+0000","last_undegraded":"2023-08-25T02:25:06.473970+0000","last_fullsized":"2023-08-25T02:25:06.473970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:32:06.548589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474291+0000","last_change":"2023-08-25T02:25:04.238646+0000","last_active":"2023-08-25T02:25:06.474291+0000","last_peered":"2023-08-25T02:25:06.474291+0000","last_clean":"2023-08-25T02:25:06.474291+0000","last_became_active":"2023-08-25T02:25:04.238252+0000","last_became_peered":"2023-08-25T02:25:04.238252+0000","last_unstale":"2023-08-25T02:25:06.474291+0000","last_undegraded":"2023-08-25T02:25:06.474291+0000","last_fullsized":"2023-08-25T02:25:06.474291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:43:32.951632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236750+0000","last_change":"2023-08-25T02:25:04.236025+0000","last_active":"2023-08-25T02:25:04.236750+0000","last_peered":"2023-08-25T02:25:04.236750+0000","last_clean":"2023-08-25T02:25:04.236750+0000","last_became_active":"2023-08-25T02:25:04.235682+0000","last_became_peered":"2023-08-25T02:25:04.235682+0000","last_unstale":"2023-08-25T02:25:04.236750+0000","last_undegraded":"2023-08-25T02:25:04.236750+0000","last_fullsized":"2023-08-25T02:25:04.236750+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:15.209256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938102+0000","last_change":"2023-08-25T02:25:04.236295+0000","last_active":"2023-08-25T02:25:05.938102+0000","last_peered":"2023-08-25T02:25:05.938102+0000","last_clean":"2023-08-25T02:25:05.938102+0000","last_became_active":"2023-08-25T02:25:04.235979+0000","last_became_peered":"2023-08-25T02:25:04.235979+0000","last_unstale":"2023-08-25T02:25:05.938102+0000","last_undegraded":"2023-08-25T02:25:05.938102+0000","last_fullsized":"2023-08-25T02:25:05.938102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:12:13.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938095+0000","last_change":"2023-08-25T02:25:04.234885+0000","last_active":"2023-08-25T02:25:05.938095+0000","last_peered":"2023-08-25T02:25:05.938095+0000","last_clean":"2023-08-25T02:25:05.938095+0000","last_became_active":"2023-08-25T02:25:04.234603+0000","last_became_peered":"2023-08-25T02:25:04.234603+0000","last_unstale":"2023-08-25T02:25:05.938095+0000","last_undegraded":"2023-08-25T02:25:05.938095+0000","last_fullsized":"2023-08-25T02:25:05.938095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:26.095674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438191+0000","last_change":"2023-08-25T02:25:04.230151+0000","last_active":"2023-08-25T02:25:06.438191+0000","last_peered":"2023-08-25T02:25:06.438191+0000","last_clean":"2023-08-25T02:25:06.438191+0000","last_became_active":"2023-08-25T02:25:04.227881+0000","last_became_peered":"2023-08-25T02:25:04.227881+0000","last_unstale":"2023-08-25T02:25:06.438191+0000","last_undegraded":"2023-08-25T02:25:06.438191+0000","last_fullsized":"2023-08-25T02:25:06.438191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:31.382761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236507+0000","last_change":"2023-08-25T02:25:04.230117+0000","last_active":"2023-08-25T02:25:04.236507+0000","last_peered":"2023-08-25T02:25:04.236507+0000","last_clean":"2023-08-25T02:25:04.236507+0000","last_became_active":"2023-08-25T02:25:04.229810+0000","last_became_peered":"2023-08-25T02:25:04.229810+0000","last_unstale":"2023-08-25T02:25:04.236507+0000","last_undegraded":"2023-08-25T02:25:04.236507+0000","last_fullsized":"2023-08-25T02:25:04.236507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:25:31.424239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.235629+0000","last_change":"2023-08-25T02:25:04.235011+0000","last_active":"2023-08-25T02:25:04.235629+0000","last_peered":"2023-08-25T02:25:04.235629+0000","last_clean":"2023-08-25T02:25:04.235629+0000","last_became_active":"2023-08-25T02:25:04.234664+0000","last_became_peered":"2023-08-25T02:25:04.234664+0000","last_unstale":"2023-08-25T02:25:04.235629+0000","last_undegraded":"2023-08-25T02:25:04.235629+0000","last_fullsized":"2023-08-25T02:25:04.235629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:22:16.915760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230565+0000","last_change":"2023-08-25T02:25:04.229223+0000","last_active":"2023-08-25T02:25:04.230565+0000","last_peered":"2023-08-25T02:25:04.230565+0000","last_clean":"2023-08-25T02:25:04.230565+0000","last_became_active":"2023-08-25T02:25:04.227656+0000","last_became_peered":"2023-08-25T02:25:04.227656+0000","last_unstale":"2023-08-25T02:25:04.230565+0000","last_undegraded":"2023-08-25T02:25:04.230565+0000","last_fullsized":"2023-08-25T02:25:04.230565+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:46:08.334390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.611544+0000","last_change":"2023-08-25T02:25:03.613357+0000","last_active":"2023-08-25T02:25:04.611544+0000","last_peered":"2023-08-25T02:25:04.611544+0000","last_clean":"2023-08-25T02:25:04.611544+0000","last_became_active":"2023-08-25T02:25:03.612977+0000","last_became_peered":"2023-08-25T02:25:03.612977+0000","last_unstale":"2023-08-25T02:25:04.611544+0000","last_undegraded":"2023-08-25T02:25:04.611544+0000","last_fullsized":"2023-08-25T02:25:04.611544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:14:21.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214090+0000","last_change":"2023-08-25T02:25:04.230544+0000","last_active":"2023-08-25T02:25:05.214090+0000","last_peered":"2023-08-25T02:25:05.214090+0000","last_clean":"2023-08-25T02:25:05.214090+0000","last_became_active":"2023-08-25T02:25:04.230226+0000","last_became_peered":"2023-08-25T02:25:04.230226+0000","last_unstale":"2023-08-25T02:25:05.214090+0000","last_undegraded":"2023-08-25T02:25:05.214090+0000","last_fullsized":"2023-08-25T02:25:05.214090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:41.647707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214191+0000","last_change":"2023-08-25T02:25:03.617626+0000","last_active":"2023-08-25T02:25:05.214191+0000","last_peered":"2023-08-25T02:25:05.214191+0000","last_clean":"2023-08-25T02:25:05.214191+0000","last_became_active":"2023-08-25T02:25:03.617323+0000","last_became_peered":"2023-08-25T02:25:03.617323+0000","last_unstale":"2023-08-25T02:25:05.214191+0000","last_undegraded":"2023-08-25T02:25:05.214191+0000","last_fullsized":"2023-08-25T02:25:05.214191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:05.360379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938528+0000","last_change":"2023-08-25T02:25:04.226808+0000","last_active":"2023-08-25T02:25:05.938528+0000","last_peered":"2023-08-25T02:25:05.938528+0000","last_clean":"2023-08-25T02:25:05.938528+0000","last_became_active":"2023-08-25T02:25:04.226407+0000","last_became_peered":"2023-08-25T02:25:04.226407+0000","last_unstale":"2023-08-25T02:25:05.938528+0000","last_undegraded":"2023-08-25T02:25:05.938528+0000","last_fullsized":"2023-08-25T02:25:05.938528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:26:23.478853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657716+0000","last_change":"2023-08-25T02:25:03.611831+0000","last_active":"2023-08-25T02:25:06.657716+0000","last_peered":"2023-08-25T02:25:06.657716+0000","last_clean":"2023-08-25T02:25:06.657716+0000","last_became_active":"2023-08-25T02:25:03.611530+0000","last_became_peered":"2023-08-25T02:25:03.611530+0000","last_unstale":"2023-08-25T02:25:06.657716+0000","last_undegraded":"2023-08-25T02:25:06.657716+0000","last_fullsized":"2023-08-25T02:25:06.657716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:02:49.263476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438337+0000","last_change":"2023-08-25T02:25:04.236572+0000","last_active":"2023-08-25T02:25:06.438337+0000","last_peered":"2023-08-25T02:25:06.438337+0000","last_clean":"2023-08-25T02:25:06.438337+0000","last_became_active":"2023-08-25T02:25:04.236242+0000","last_became_peered":"2023-08-25T02:25:04.236242+0000","last_unstale":"2023-08-25T02:25:06.438337+0000","last_undegraded":"2023-08-25T02:25:06.438337+0000","last_fullsized":"2023-08-25T02:25:06.438337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:45:50.267973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657579+0000","last_change":"2023-08-25T02:25:03.616265+0000","last_active":"2023-08-25T02:25:06.657579+0000","last_peered":"2023-08-25T02:25:06.657579+0000","last_clean":"2023-08-25T02:25:06.657579+0000","last_became_active":"2023-08-25T02:25:03.615926+0000","last_became_peered":"2023-08-25T02:25:03.615926+0000","last_unstale":"2023-08-25T02:25:06.657579+0000","last_undegraded":"2023-08-25T02:25:06.657579+0000","last_fullsized":"2023-08-25T02:25:06.657579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:21:34.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157129+0000","last_change":"2023-08-25T02:25:04.232925+0000","last_active":"2023-08-25T02:25:06.157129+0000","last_peered":"2023-08-25T02:25:06.157129+0000","last_clean":"2023-08-25T02:25:06.157129+0000","last_became_active":"2023-08-25T02:25:04.232615+0000","last_became_peered":"2023-08-25T02:25:04.232615+0000","last_unstale":"2023-08-25T02:25:06.157129+0000","last_undegraded":"2023-08-25T02:25:06.157129+0000","last_fullsized":"2023-08-25T02:25:06.157129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:48:54.025387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633789+0000","last_change":"2023-08-25T02:25:03.618534+0000","last_active":"2023-08-25T02:25:07.633789+0000","last_peered":"2023-08-25T02:25:07.633789+0000","last_clean":"2023-08-25T02:25:07.633789+0000","last_became_active":"2023-08-25T02:25:03.618284+0000","last_became_peered":"2023-08-25T02:25:03.618284+0000","last_unstale":"2023-08-25T02:25:07.633789+0000","last_undegraded":"2023-08-25T02:25:07.633789+0000","last_fullsized":"2023-08-25T02:25:07.633789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:55:34.354817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657492+0000","last_change":"2023-08-25T02:25:04.225148+0000","last_active":"2023-08-25T02:25:06.657492+0000","last_peered":"2023-08-25T02:25:06.657492+0000","last_clean":"2023-08-25T02:25:06.657492+0000","last_became_active":"2023-08-25T02:25:04.224776+0000","last_became_peered":"2023-08-25T02:25:04.224776+0000","last_unstale":"2023-08-25T02:25:06.657492+0000","last_undegraded":"2023-08-25T02:25:06.657492+0000","last_fullsized":"2023-08-25T02:25:06.657492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:27.220482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214314+0000","last_change":"2023-08-25T02:25:03.610869+0000","last_active":"2023-08-25T02:25:05.214314+0000","last_peered":"2023-08-25T02:25:05.214314+0000","last_clean":"2023-08-25T02:25:05.214314+0000","last_became_active":"2023-08-25T02:25:03.610568+0000","last_became_peered":"2023-08-25T02:25:03.610568+0000","last_unstale":"2023-08-25T02:25:05.214314+0000","last_undegraded":"2023-08-25T02:25:05.214314+0000","last_fullsized":"2023-08-25T02:25:05.214314+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:16.686802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157177+0000","last_change":"2023-08-25T02:25:04.230241+0000","last_active":"2023-08-25T02:25:06.157177+0000","last_peered":"2023-08-25T02:25:06.157177+0000","last_clean":"2023-08-25T02:25:06.157177+0000","last_became_active":"2023-08-25T02:25:04.229871+0000","last_became_peered":"2023-08-25T02:25:04.229871+0000","last_unstale":"2023-08-25T02:25:06.157177+0000","last_undegraded":"2023-08-25T02:25:06.157177+0000","last_fullsized":"2023-08-25T02:25:06.157177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:20:16.764817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.217198+0000","last_change":"2023-08-25T02:25:03.610650+0000","last_active":"2023-08-25T02:25:04.217198+0000","last_peered":"2023-08-25T02:25:04.217198+0000","last_clean":"2023-08-25T02:25:04.217198+0000","last_became_active":"2023-08-25T02:25:03.609901+0000","last_became_peered":"2023-08-25T02:25:03.609901+0000","last_unstale":"2023-08-25T02:25:04.217198+0000","last_undegraded":"2023-08-25T02:25:04.217198+0000","last_fullsized":"2023-08-25T02:25:04.217198+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:58.744550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213759+0000","last_change":"2023-08-25T02:25:04.222682+0000","last_active":"2023-08-25T02:25:05.213759+0000","last_peered":"2023-08-25T02:25:05.213759+0000","last_clean":"2023-08-25T02:25:05.213759+0000","last_became_active":"2023-08-25T02:25:04.221791+0000","last_became_peered":"2023-08-25T02:25:04.221791+0000","last_unstale":"2023-08-25T02:25:05.213759+0000","last_undegraded":"2023-08-25T02:25:05.213759+0000","last_fullsized":"2023-08-25T02:25:05.213759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:29:20.943592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156809+0000","last_change":"2023-08-25T02:25:03.610342+0000","last_active":"2023-08-25T02:25:06.156809+0000","last_peered":"2023-08-25T02:25:06.156809+0000","last_clean":"2023-08-25T02:25:06.156809+0000","last_became_active":"2023-08-25T02:25:03.609928+0000","last_became_peered":"2023-08-25T02:25:03.609928+0000","last_unstale":"2023-08-25T02:25:06.156809+0000","last_undegraded":"2023-08-25T02:25:06.156809+0000","last_fullsized":"2023-08-25T02:25:06.156809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:42.862108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.235064+0000","last_change":"2023-08-25T02:25:04.234364+0000","last_active":"2023-08-25T02:25:04.235064+0000","last_peered":"2023-08-25T02:25:04.235064+0000","last_clean":"2023-08-25T02:25:04.235064+0000","last_became_active":"2023-08-25T02:25:04.234050+0000","last_became_peered":"2023-08-25T02:25:04.234050+0000","last_unstale":"2023-08-25T02:25:04.235064+0000","last_undegraded":"2023-08-25T02:25:04.235064+0000","last_fullsized":"2023-08-25T02:25:04.235064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:18:36.177255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214477+0000","last_change":"2023-08-25T02:25:03.617016+0000","last_active":"2023-08-25T02:25:05.214477+0000","last_peered":"2023-08-25T02:25:05.214477+0000","last_clean":"2023-08-25T02:25:05.214477+0000","last_became_active":"2023-08-25T02:25:03.616670+0000","last_became_peered":"2023-08-25T02:25:03.616670+0000","last_unstale":"2023-08-25T02:25:05.214477+0000","last_undegraded":"2023-08-25T02:25:05.214477+0000","last_fullsized":"2023-08-25T02:25:05.214477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:42:03.302702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438625+0000","last_change":"2023-08-25T02:25:04.236009+0000","last_active":"2023-08-25T02:25:06.438625+0000","last_peered":"2023-08-25T02:25:06.438625+0000","last_clean":"2023-08-25T02:25:06.438625+0000","last_became_active":"2023-08-25T02:25:04.235722+0000","last_became_peered":"2023-08-25T02:25:04.235722+0000","last_unstale":"2023-08-25T02:25:06.438625+0000","last_undegraded":"2023-08-25T02:25:06.438625+0000","last_fullsized":"2023-08-25T02:25:06.438625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:42:44.656781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474018+0000","last_change":"2023-08-25T02:25:03.611060+0000","last_active":"2023-08-25T02:25:06.474018+0000","last_peered":"2023-08-25T02:25:06.474018+0000","last_clean":"2023-08-25T02:25:06.474018+0000","last_became_active":"2023-08-25T02:25:03.610715+0000","last_became_peered":"2023-08-25T02:25:03.610715+0000","last_unstale":"2023-08-25T02:25:06.474018+0000","last_undegraded":"2023-08-25T02:25:06.474018+0000","last_fullsized":"2023-08-25T02:25:06.474018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:17.684716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938519+0000","last_change":"2023-08-25T02:25:04.225855+0000","last_active":"2023-08-25T02:25:05.938519+0000","last_peered":"2023-08-25T02:25:05.938519+0000","last_clean":"2023-08-25T02:25:05.938519+0000","last_became_active":"2023-08-25T02:25:04.225559+0000","last_became_peered":"2023-08-25T02:25:04.225559+0000","last_unstale":"2023-08-25T02:25:05.938519+0000","last_undegraded":"2023-08-25T02:25:05.938519+0000","last_fullsized":"2023-08-25T02:25:05.938519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:33:19.054268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157230+0000","last_change":"2023-08-25T02:25:03.617097+0000","last_active":"2023-08-25T02:25:06.157230+0000","last_peered":"2023-08-25T02:25:06.157230+0000","last_clean":"2023-08-25T02:25:06.157230+0000","last_became_active":"2023-08-25T02:25:03.616809+0000","last_became_peered":"2023-08-25T02:25:03.616809+0000","last_unstale":"2023-08-25T02:25:06.157230+0000","last_undegraded":"2023-08-25T02:25:06.157230+0000","last_fullsized":"2023-08-25T02:25:06.157230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:11:24.908899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438483+0000","last_change":"2023-08-25T02:25:04.226384+0000","last_active":"2023-08-25T02:25:06.438483+0000","last_peered":"2023-08-25T02:25:06.438483+0000","last_clean":"2023-08-25T02:25:06.438483+0000","last_became_active":"2023-08-25T02:25:04.226047+0000","last_became_peered":"2023-08-25T02:25:04.226047+0000","last_unstale":"2023-08-25T02:25:06.438483+0000","last_undegraded":"2023-08-25T02:25:06.438483+0000","last_fullsized":"2023-08-25T02:25:06.438483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:01.485924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214517+0000","last_change":"2023-08-25T02:25:03.609570+0000","last_active":"2023-08-25T02:25:05.214517+0000","last_peered":"2023-08-25T02:25:05.214517+0000","last_clean":"2023-08-25T02:25:05.214517+0000","last_became_active":"2023-08-25T02:25:03.609253+0000","last_became_peered":"2023-08-25T02:25:03.609253+0000","last_unstale":"2023-08-25T02:25:05.214517+0000","last_undegraded":"2023-08-25T02:25:05.214517+0000","last_fullsized":"2023-08-25T02:25:05.214517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:14.775308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657588+0000","last_change":"2023-08-25T02:25:04.226667+0000","last_active":"2023-08-25T02:25:06.657588+0000","last_peered":"2023-08-25T02:25:06.657588+0000","last_clean":"2023-08-25T02:25:06.657588+0000","last_became_active":"2023-08-25T02:25:04.226332+0000","last_became_peered":"2023-08-25T02:25:04.226332+0000","last_unstale":"2023-08-25T02:25:06.657588+0000","last_undegraded":"2023-08-25T02:25:06.657588+0000","last_fullsized":"2023-08-25T02:25:06.657588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:51:57.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.610718+0000","last_change":"2023-08-25T02:25:03.609419+0000","last_active":"2023-08-25T02:25:04.610718+0000","last_peered":"2023-08-25T02:25:04.610718+0000","last_clean":"2023-08-25T02:25:04.610718+0000","last_became_active":"2023-08-25T02:25:03.609101+0000","last_became_peered":"2023-08-25T02:25:03.609101+0000","last_unstale":"2023-08-25T02:25:04.610718+0000","last_undegraded":"2023-08-25T02:25:04.610718+0000","last_fullsized":"2023-08-25T02:25:04.610718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:54:48.243106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232161+0000","last_change":"2023-08-25T02:25:04.231351+0000","last_active":"2023-08-25T02:25:04.232161+0000","last_peered":"2023-08-25T02:25:04.232161+0000","last_clean":"2023-08-25T02:25:04.232161+0000","last_became_active":"2023-08-25T02:25:04.230974+0000","last_became_peered":"2023-08-25T02:25:04.230974+0000","last_unstale":"2023-08-25T02:25:04.232161+0000","last_undegraded":"2023-08-25T02:25:04.232161+0000","last_fullsized":"2023-08-25T02:25:04.232161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:55:37.390357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633217+0000","last_change":"2023-08-25T02:25:03.617889+0000","last_active":"2023-08-25T02:25:07.633217+0000","last_peered":"2023-08-25T02:25:07.633217+0000","last_clean":"2023-08-25T02:25:07.633217+0000","last_became_active":"2023-08-25T02:25:03.617576+0000","last_became_peered":"2023-08-25T02:25:03.617576+0000","last_unstale":"2023-08-25T02:25:07.633217+0000","last_undegraded":"2023-08-25T02:25:07.633217+0000","last_fullsized":"2023-08-25T02:25:07.633217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:38:22.627438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473924+0000","last_change":"2023-08-25T02:25:04.230135+0000","last_active":"2023-08-25T02:25:06.473924+0000","last_peered":"2023-08-25T02:25:06.473924+0000","last_clean":"2023-08-25T02:25:06.473924+0000","last_became_active":"2023-08-25T02:25:04.229592+0000","last_became_peered":"2023-08-25T02:25:04.229592+0000","last_unstale":"2023-08-25T02:25:06.473924+0000","last_undegraded":"2023-08-25T02:25:06.473924+0000","last_fullsized":"2023-08-25T02:25:06.473924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:28:47.602421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473908+0000","last_change":"2023-08-25T02:25:03.610021+0000","last_active":"2023-08-25T02:25:06.473908+0000","last_peered":"2023-08-25T02:25:06.473908+0000","last_clean":"2023-08-25T02:25:06.473908+0000","last_became_active":"2023-08-25T02:25:03.609703+0000","last_became_peered":"2023-08-25T02:25:03.609703+0000","last_unstale":"2023-08-25T02:25:06.473908+0000","last_undegraded":"2023-08-25T02:25:06.473908+0000","last_fullsized":"2023-08-25T02:25:06.473908+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:25.169266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473493+0000","last_change":"2023-08-25T02:25:04.234643+0000","last_active":"2023-08-25T02:25:06.473493+0000","last_peered":"2023-08-25T02:25:06.473493+0000","last_clean":"2023-08-25T02:25:06.473493+0000","last_became_active":"2023-08-25T02:25:04.234349+0000","last_became_peered":"2023-08-25T02:25:04.234349+0000","last_unstale":"2023-08-25T02:25:06.473493+0000","last_undegraded":"2023-08-25T02:25:06.473493+0000","last_fullsized":"2023-08-25T02:25:06.473493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:08:25.591726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'10","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632840+0000","last_change":"2023-08-25T02:25:03.613702+0000","last_active":"2023-08-25T02:25:07.632840+0000","last_peered":"2023-08-25T02:25:07.632840+0000","last_clean":"2023-08-25T02:25:07.632840+0000","last_became_active":"2023-08-25T02:25:03.613383+0000","last_became_peered":"2023-08-25T02:25:03.613383+0000","last_unstale":"2023-08-25T02:25:07.632840+0000","last_undegraded":"2023-08-25T02:25:07.632840+0000","last_fullsized":"2023-08-25T02:25:07.632840+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:53.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156815+0000","last_change":"2023-08-25T02:25:04.227650+0000","last_active":"2023-08-25T02:25:06.156815+0000","last_peered":"2023-08-25T02:25:06.156815+0000","last_clean":"2023-08-25T02:25:06.156815+0000","last_became_active":"2023-08-25T02:25:04.227335+0000","last_became_peered":"2023-08-25T02:25:04.227335+0000","last_unstale":"2023-08-25T02:25:06.156815+0000","last_undegraded":"2023-08-25T02:25:06.156815+0000","last_fullsized":"2023-08-25T02:25:06.156815+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:10.086931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657623+0000","last_change":"2023-08-25T02:25:03.616065+0000","last_active":"2023-08-25T02:25:06.657623+0000","last_peered":"2023-08-25T02:25:06.657623+0000","last_clean":"2023-08-25T02:25:06.657623+0000","last_became_active":"2023-08-25T02:25:03.615751+0000","last_became_peered":"2023-08-25T02:25:03.615751+0000","last_unstale":"2023-08-25T02:25:06.657623+0000","last_undegraded":"2023-08-25T02:25:06.657623+0000","last_fullsized":"2023-08-25T02:25:06.657623+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:27:27.917233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230930+0000","last_change":"2023-08-25T02:25:04.228285+0000","last_active":"2023-08-25T02:25:04.230930+0000","last_peered":"2023-08-25T02:25:04.230930+0000","last_clean":"2023-08-25T02:25:04.230930+0000","last_became_active":"2023-08-25T02:25:04.228061+0000","last_became_peered":"2023-08-25T02:25:04.228061+0000","last_unstale":"2023-08-25T02:25:04.230930+0000","last_undegraded":"2023-08-25T02:25:04.230930+0000","last_fullsized":"2023-08-25T02:25:04.230930+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:41:57.051512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.218918+0000","last_change":"2023-08-25T02:25:03.615712+0000","last_active":"2023-08-25T02:25:04.218918+0000","last_peered":"2023-08-25T02:25:04.218918+0000","last_clean":"2023-08-25T02:25:04.218918+0000","last_became_active":"2023-08-25T02:25:03.615412+0000","last_became_peered":"2023-08-25T02:25:03.615412+0000","last_unstale":"2023-08-25T02:25:04.218918+0000","last_undegraded":"2023-08-25T02:25:04.218918+0000","last_fullsized":"2023-08-25T02:25:04.218918+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:15.238634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214408+0000","last_change":"2023-08-25T02:25:04.233485+0000","last_active":"2023-08-25T02:25:05.214408+0000","last_peered":"2023-08-25T02:25:05.214408+0000","last_clean":"2023-08-25T02:25:05.214408+0000","last_became_active":"2023-08-25T02:25:04.233205+0000","last_became_peered":"2023-08-25T02:25:04.233205+0000","last_unstale":"2023-08-25T02:25:05.214408+0000","last_undegraded":"2023-08-25T02:25:05.214408+0000","last_fullsized":"2023-08-25T02:25:05.214408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:12.219109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473945+0000","last_change":"2023-08-25T02:25:03.613848+0000","last_active":"2023-08-25T02:25:06.473945+0000","last_peered":"2023-08-25T02:25:06.473945+0000","last_clean":"2023-08-25T02:25:06.473945+0000","last_became_active":"2023-08-25T02:25:03.613538+0000","last_became_peered":"2023-08-25T02:25:03.613538+0000","last_unstale":"2023-08-25T02:25:06.473945+0000","last_undegraded":"2023-08-25T02:25:06.473945+0000","last_fullsized":"2023-08-25T02:25:06.473945+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:52.883965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214475+0000","last_change":"2023-08-25T02:25:04.234167+0000","last_active":"2023-08-25T02:25:05.214475+0000","last_peered":"2023-08-25T02:25:05.214475+0000","last_clean":"2023-08-25T02:25:05.214475+0000","last_became_active":"2023-08-25T02:25:04.233779+0000","last_became_peered":"2023-08-25T02:25:04.233779+0000","last_unstale":"2023-08-25T02:25:05.214475+0000","last_undegraded":"2023-08-25T02:25:05.214475+0000","last_fullsized":"2023-08-25T02:25:05.214475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:19.007690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156909+0000","last_change":"2023-08-25T02:25:04.227759+0000","last_active":"2023-08-25T02:25:06.156909+0000","last_peered":"2023-08-25T02:25:06.156909+0000","last_clean":"2023-08-25T02:25:06.156909+0000","last_became_active":"2023-08-25T02:25:04.227166+0000","last_became_peered":"2023-08-25T02:25:04.227166+0000","last_unstale":"2023-08-25T02:25:06.156909+0000","last_undegraded":"2023-08-25T02:25:06.156909+0000","last_fullsized":"2023-08-25T02:25:06.156909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:58.916927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.217095+0000","last_change":"2023-08-25T02:25:03.613384+0000","last_active":"2023-08-25T02:25:04.217095+0000","last_peered":"2023-08-25T02:25:04.217095+0000","last_clean":"2023-08-25T02:25:04.217095+0000","last_became_active":"2023-08-25T02:25:03.613097+0000","last_became_peered":"2023-08-25T02:25:03.613097+0000","last_unstale":"2023-08-25T02:25:04.217095+0000","last_undegraded":"2023-08-25T02:25:04.217095+0000","last_fullsized":"2023-08-25T02:25:04.217095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:43:03.447001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214227+0000","last_change":"2023-08-25T02:25:04.225009+0000","last_active":"2023-08-25T02:25:05.214227+0000","last_peered":"2023-08-25T02:25:05.214227+0000","last_clean":"2023-08-25T02:25:05.214227+0000","last_became_active":"2023-08-25T02:25:04.224678+0000","last_became_peered":"2023-08-25T02:25:04.224678+0000","last_unstale":"2023-08-25T02:25:05.214227+0000","last_undegraded":"2023-08-25T02:25:05.214227+0000","last_fullsized":"2023-08-25T02:25:05.214227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:45:44.498691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632106+0000","last_change":"2023-08-25T02:25:03.607620+0000","last_active":"2023-08-25T02:25:07.632106+0000","last_peered":"2023-08-25T02:25:07.632106+0000","last_clean":"2023-08-25T02:25:07.632106+0000","last_became_active":"2023-08-25T02:25:03.607329+0000","last_became_peered":"2023-08-25T02:25:03.607329+0000","last_unstale":"2023-08-25T02:25:07.632106+0000","last_undegraded":"2023-08-25T02:25:07.632106+0000","last_fullsized":"2023-08-25T02:25:07.632106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:21.994627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473471+0000","last_change":"2023-08-25T02:24:05.131624+0000","last_active":"2023-08-25T02:25:06.473471+0000","last_peered":"2023-08-25T02:25:06.473471+0000","last_clean":"2023-08-25T02:25:06.473471+0000","last_became_active":"2023-08-25T02:24:05.131084+0000","last_became_peered":"2023-08-25T02:24:05.131084+0000","last_unstale":"2023-08-25T02:25:06.473471+0000","last_undegraded":"2023-08-25T02:25:06.473471+0000","last_fullsized":"2023-08-25T02:25:06.473471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_clean_scrub_stamp":"2023-08-25T02:21:10.049896+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:44.618723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232268+0000","last_change":"2023-08-25T02:25:04.231797+0000","last_active":"2023-08-25T02:25:04.232268+0000","last_peered":"2023-08-25T02:25:04.232268+0000","last_clean":"2023-08-25T02:25:04.232268+0000","last_became_active":"2023-08-25T02:25:04.231520+0000","last_became_peered":"2023-08-25T02:25:04.231520+0000","last_unstale":"2023-08-25T02:25:04.232268+0000","last_undegraded":"2023-08-25T02:25:04.232268+0000","last_fullsized":"2023-08-25T02:25:04.232268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:55:20.388497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473636+0000","last_change":"2023-08-25T02:25:03.611517+0000","last_active":"2023-08-25T02:25:06.473636+0000","last_peered":"2023-08-25T02:25:06.473636+0000","last_clean":"2023-08-25T02:25:06.473636+0000","last_became_active":"2023-08-25T02:25:03.611205+0000","last_became_peered":"2023-08-25T02:25:03.611205+0000","last_unstale":"2023-08-25T02:25:06.473636+0000","last_undegraded":"2023-08-25T02:25:06.473636+0000","last_fullsized":"2023-08-25T02:25:06.473636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:04:13.071834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657665+0000","last_change":"2023-08-25T02:25:03.613795+0000","last_active":"2023-08-25T02:25:06.657665+0000","last_peered":"2023-08-25T02:25:06.657665+0000","last_clean":"2023-08-25T02:25:06.657665+0000","last_became_active":"2023-08-25T02:25:03.613410+0000","last_became_peered":"2023-08-25T02:25:03.613410+0000","last_unstale":"2023-08-25T02:25:06.657665+0000","last_undegraded":"2023-08-25T02:25:06.657665+0000","last_fullsized":"2023-08-25T02:25:06.657665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:47:37.646320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232803+0000","last_change":"2023-08-25T02:25:04.232186+0000","last_active":"2023-08-25T02:25:04.232803+0000","last_peered":"2023-08-25T02:25:04.232803+0000","last_clean":"2023-08-25T02:25:04.232803+0000","last_became_active":"2023-08-25T02:25:04.231853+0000","last_became_peered":"2023-08-25T02:25:04.231853+0000","last_unstale":"2023-08-25T02:25:04.232803+0000","last_undegraded":"2023-08-25T02:25:04.232803+0000","last_fullsized":"2023-08-25T02:25:04.232803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:14:44.584942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.219541+0000","last_change":"2023-08-25T02:25:03.615541+0000","last_active":"2023-08-25T02:25:04.219541+0000","last_peered":"2023-08-25T02:25:04.219541+0000","last_clean":"2023-08-25T02:25:04.219541+0000","last_became_active":"2023-08-25T02:25:03.615196+0000","last_became_peered":"2023-08-25T02:25:03.615196+0000","last_unstale":"2023-08-25T02:25:04.219541+0000","last_undegraded":"2023-08-25T02:25:04.219541+0000","last_fullsized":"2023-08-25T02:25:04.219541+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:06:02.183736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214288+0000","last_change":"2023-08-25T02:25:04.227688+0000","last_active":"2023-08-25T02:25:05.214288+0000","last_peered":"2023-08-25T02:25:05.214288+0000","last_clean":"2023-08-25T02:25:05.214288+0000","last_became_active":"2023-08-25T02:25:04.227394+0000","last_became_peered":"2023-08-25T02:25:04.227394+0000","last_unstale":"2023-08-25T02:25:05.214288+0000","last_undegraded":"2023-08-25T02:25:05.214288+0000","last_fullsized":"2023-08-25T02:25:05.214288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:53:57.237749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633564+0000","last_change":"2023-08-25T02:25:03.608500+0000","last_active":"2023-08-25T02:25:07.633564+0000","last_peered":"2023-08-25T02:25:07.633564+0000","last_clean":"2023-08-25T02:25:07.633564+0000","last_became_active":"2023-08-25T02:25:03.608172+0000","last_became_peered":"2023-08-25T02:25:03.608172+0000","last_unstale":"2023-08-25T02:25:07.633564+0000","last_undegraded":"2023-08-25T02:25:07.633564+0000","last_fullsized":"2023-08-25T02:25:07.633564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:18:45.063308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938372+0000","last_change":"2023-08-25T02:25:04.226768+0000","last_active":"2023-08-25T02:25:05.938372+0000","last_peered":"2023-08-25T02:25:05.938372+0000","last_clean":"2023-08-25T02:25:05.938372+0000","last_became_active":"2023-08-25T02:25:04.226459+0000","last_became_peered":"2023-08-25T02:25:04.226459+0000","last_unstale":"2023-08-25T02:25:05.938372+0000","last_undegraded":"2023-08-25T02:25:05.938372+0000","last_fullsized":"2023-08-25T02:25:05.938372+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:04:52.374035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438230+0000","last_change":"2023-08-25T02:25:03.612306+0000","last_active":"2023-08-25T02:25:06.438230+0000","last_peered":"2023-08-25T02:25:06.438230+0000","last_clean":"2023-08-25T02:25:06.438230+0000","last_became_active":"2023-08-25T02:25:03.611922+0000","last_became_peered":"2023-08-25T02:25:03.611922+0000","last_unstale":"2023-08-25T02:25:06.438230+0000","last_undegraded":"2023-08-25T02:25:06.438230+0000","last_fullsized":"2023-08-25T02:25:06.438230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:14:09.728422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657329+0000","last_change":"2023-08-25T02:25:04.235032+0000","last_active":"2023-08-25T02:25:06.657329+0000","last_peered":"2023-08-25T02:25:06.657329+0000","last_clean":"2023-08-25T02:25:06.657329+0000","last_became_active":"2023-08-25T02:25:04.234626+0000","last_became_peered":"2023-08-25T02:25:04.234626+0000","last_unstale":"2023-08-25T02:25:06.657329+0000","last_undegraded":"2023-08-25T02:25:06.657329+0000","last_fullsized":"2023-08-25T02:25:06.657329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:00.849934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213814+0000","last_change":"2023-08-25T02:25:03.608869+0000","last_active":"2023-08-25T02:25:05.213814+0000","last_peered":"2023-08-25T02:25:05.213814+0000","last_clean":"2023-08-25T02:25:05.213814+0000","last_became_active":"2023-08-25T02:25:03.608530+0000","last_became_peered":"2023-08-25T02:25:03.608530+0000","last_unstale":"2023-08-25T02:25:05.213814+0000","last_undegraded":"2023-08-25T02:25:05.213814+0000","last_fullsized":"2023-08-25T02:25:05.213814+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:51:39.415744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236891+0000","last_change":"2023-08-25T02:25:04.231830+0000","last_active":"2023-08-25T02:25:04.236891+0000","last_peered":"2023-08-25T02:25:04.236891+0000","last_clean":"2023-08-25T02:25:04.236891+0000","last_became_active":"2023-08-25T02:25:04.231530+0000","last_became_peered":"2023-08-25T02:25:04.231530+0000","last_unstale":"2023-08-25T02:25:04.236891+0000","last_undegraded":"2023-08-25T02:25:04.236891+0000","last_fullsized":"2023-08-25T02:25:04.236891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:04.520766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.220091+0000","last_change":"2023-08-25T02:25:03.610625+0000","last_active":"2023-08-25T02:25:04.220091+0000","last_peered":"2023-08-25T02:25:04.220091+0000","last_clean":"2023-08-25T02:25:04.220091+0000","last_became_active":"2023-08-25T02:25:03.610305+0000","last_became_peered":"2023-08-25T02:25:03.610305+0000","last_unstale":"2023-08-25T02:25:04.220091+0000","last_undegraded":"2023-08-25T02:25:04.220091+0000","last_fullsized":"2023-08-25T02:25:04.220091+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:41:23.669327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214041+0000","last_change":"2023-08-25T02:25:04.234533+0000","last_active":"2023-08-25T02:25:05.214041+0000","last_peered":"2023-08-25T02:25:05.214041+0000","last_clean":"2023-08-25T02:25:05.214041+0000","last_became_active":"2023-08-25T02:25:04.234167+0000","last_became_peered":"2023-08-25T02:25:04.234167+0000","last_unstale":"2023-08-25T02:25:05.214041+0000","last_undegraded":"2023-08-25T02:25:05.214041+0000","last_fullsized":"2023-08-25T02:25:05.214041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:23:55.422360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633060+0000","last_change":"2023-08-25T02:25:03.610139+0000","last_active":"2023-08-25T02:25:07.633060+0000","last_peered":"2023-08-25T02:25:07.633060+0000","last_clean":"2023-08-25T02:25:07.633060+0000","last_became_active":"2023-08-25T02:25:03.609811+0000","last_became_peered":"2023-08-25T02:25:03.609811+0000","last_unstale":"2023-08-25T02:25:07.633060+0000","last_undegraded":"2023-08-25T02:25:07.633060+0000","last_fullsized":"2023-08-25T02:25:07.633060+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:01.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474201+0000","last_change":"2023-08-25T02:25:04.233179+0000","last_active":"2023-08-25T02:25:06.474201+0000","last_peered":"2023-08-25T02:25:06.474201+0000","last_clean":"2023-08-25T02:25:06.474201+0000","last_became_active":"2023-08-25T02:25:04.232874+0000","last_became_peered":"2023-08-25T02:25:04.232874+0000","last_unstale":"2023-08-25T02:25:06.474201+0000","last_undegraded":"2023-08-25T02:25:06.474201+0000","last_fullsized":"2023-08-25T02:25:06.474201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:54.317603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.220736+0000","last_change":"2023-08-25T02:25:03.619258+0000","last_active":"2023-08-25T02:25:04.220736+0000","last_peered":"2023-08-25T02:25:04.220736+0000","last_clean":"2023-08-25T02:25:04.220736+0000","last_became_active":"2023-08-25T02:25:03.618954+0000","last_became_peered":"2023-08-25T02:25:03.618954+0000","last_unstale":"2023-08-25T02:25:04.220736+0000","last_undegraded":"2023-08-25T02:25:04.220736+0000","last_fullsized":"2023-08-25T02:25:04.220736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:20.995650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474059+0000","last_change":"2023-08-25T02:25:04.232620+0000","last_active":"2023-08-25T02:25:06.474059+0000","last_peered":"2023-08-25T02:25:06.474059+0000","last_clean":"2023-08-25T02:25:06.474059+0000","last_became_active":"2023-08-25T02:25:04.232314+0000","last_became_peered":"2023-08-25T02:25:04.232314+0000","last_unstale":"2023-08-25T02:25:06.474059+0000","last_undegraded":"2023-08-25T02:25:06.474059+0000","last_fullsized":"2023-08-25T02:25:06.474059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:39:46.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157074+0000","last_change":"2023-08-25T02:25:03.617794+0000","last_active":"2023-08-25T02:25:06.157074+0000","last_peered":"2023-08-25T02:25:06.157074+0000","last_clean":"2023-08-25T02:25:06.157074+0000","last_became_active":"2023-08-25T02:25:03.617521+0000","last_became_peered":"2023-08-25T02:25:03.617521+0000","last_unstale":"2023-08-25T02:25:06.157074+0000","last_undegraded":"2023-08-25T02:25:06.157074+0000","last_fullsized":"2023-08-25T02:25:06.157074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:35:22.049419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474136+0000","last_change":"2023-08-25T02:25:04.229796+0000","last_active":"2023-08-25T02:25:06.474136+0000","last_peered":"2023-08-25T02:25:06.474136+0000","last_clean":"2023-08-25T02:25:06.474136+0000","last_became_active":"2023-08-25T02:25:04.227073+0000","last_became_peered":"2023-08-25T02:25:04.227073+0000","last_unstale":"2023-08-25T02:25:06.474136+0000","last_undegraded":"2023-08-25T02:25:06.474136+0000","last_fullsized":"2023-08-25T02:25:06.474136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:06:43.826591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.609293+0000","last_change":"2023-08-25T02:25:03.612702+0000","last_active":"2023-08-25T02:25:04.609293+0000","last_peered":"2023-08-25T02:25:04.609293+0000","last_clean":"2023-08-25T02:25:04.609293+0000","last_became_active":"2023-08-25T02:25:03.612482+0000","last_became_peered":"2023-08-25T02:25:03.612482+0000","last_unstale":"2023-08-25T02:25:04.609293+0000","last_undegraded":"2023-08-25T02:25:04.609293+0000","last_fullsized":"2023-08-25T02:25:04.609293+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:48.686491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474025+0000","last_change":"2023-08-25T02:25:04.229892+0000","last_active":"2023-08-25T02:25:06.474025+0000","last_peered":"2023-08-25T02:25:06.474025+0000","last_clean":"2023-08-25T02:25:06.474025+0000","last_became_active":"2023-08-25T02:25:04.229581+0000","last_became_peered":"2023-08-25T02:25:04.229581+0000","last_unstale":"2023-08-25T02:25:06.474025+0000","last_undegraded":"2023-08-25T02:25:06.474025+0000","last_fullsized":"2023-08-25T02:25:06.474025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:15:54.252164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236914+0000","last_change":"2023-08-25T02:25:04.231992+0000","last_active":"2023-08-25T02:25:04.236914+0000","last_peered":"2023-08-25T02:25:04.236914+0000","last_clean":"2023-08-25T02:25:04.236914+0000","last_became_active":"2023-08-25T02:25:04.231676+0000","last_became_peered":"2023-08-25T02:25:04.231676+0000","last_unstale":"2023-08-25T02:25:04.236914+0000","last_undegraded":"2023-08-25T02:25:04.236914+0000","last_fullsized":"2023-08-25T02:25:04.236914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:48:53.083401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230592+0000","last_change":"2023-08-25T02:25:04.227123+0000","last_active":"2023-08-25T02:25:04.230592+0000","last_peered":"2023-08-25T02:25:04.230592+0000","last_clean":"2023-08-25T02:25:04.230592+0000","last_became_active":"2023-08-25T02:25:04.226828+0000","last_became_peered":"2023-08-25T02:25:04.226828+0000","last_unstale":"2023-08-25T02:25:04.230592+0000","last_undegraded":"2023-08-25T02:25:04.230592+0000","last_fullsized":"2023-08-25T02:25:04.230592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:59:31.717429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.233062+0000","last_change":"2023-08-25T02:25:04.232635+0000","last_active":"2023-08-25T02:25:04.233062+0000","last_peered":"2023-08-25T02:25:04.233062+0000","last_clean":"2023-08-25T02:25:04.233062+0000","last_became_active":"2023-08-25T02:25:04.232353+0000","last_became_peered":"2023-08-25T02:25:04.232353+0000","last_unstale":"2023-08-25T02:25:04.233062+0000","last_undegraded":"2023-08-25T02:25:04.233062+0000","last_fullsized":"2023-08-25T02:25:04.233062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:54:26.193989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938254+0000","last_change":"2023-08-25T02:25:04.226977+0000","last_active":"2023-08-25T02:25:05.938254+0000","last_peered":"2023-08-25T02:25:05.938254+0000","last_clean":"2023-08-25T02:25:05.938254+0000","last_became_active":"2023-08-25T02:25:04.226579+0000","last_became_peered":"2023-08-25T02:25:04.226579+0000","last_unstale":"2023-08-25T02:25:05.938254+0000","last_undegraded":"2023-08-25T02:25:05.938254+0000","last_fullsized":"2023-08-25T02:25:05.938254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:00.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.225841+0000","last_change":"2023-08-25T02:25:04.225050+0000","last_active":"2023-08-25T02:25:04.225841+0000","last_peered":"2023-08-25T02:25:04.225841+0000","last_clean":"2023-08-25T02:25:04.225841+0000","last_became_active":"2023-08-25T02:25:04.224720+0000","last_became_peered":"2023-08-25T02:25:04.224720+0000","last_unstale":"2023-08-25T02:25:04.225841+0000","last_undegraded":"2023-08-25T02:25:04.225841+0000","last_fullsized":"2023-08-25T02:25:04.225841+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:29.863461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230964+0000","last_change":"2023-08-25T02:25:04.228996+0000","last_active":"2023-08-25T02:25:04.230964+0000","last_peered":"2023-08-25T02:25:04.230964+0000","last_clean":"2023-08-25T02:25:04.230964+0000","last_became_active":"2023-08-25T02:25:04.228526+0000","last_became_peered":"2023-08-25T02:25:04.228526+0000","last_unstale":"2023-08-25T02:25:04.230964+0000","last_undegraded":"2023-08-25T02:25:04.230964+0000","last_fullsized":"2023-08-25T02:25:04.230964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:00:39.738094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236090+0000","last_change":"2023-08-25T02:25:04.234921+0000","last_active":"2023-08-25T02:25:04.236090+0000","last_peered":"2023-08-25T02:25:04.236090+0000","last_clean":"2023-08-25T02:25:04.236090+0000","last_became_active":"2023-08-25T02:25:04.234588+0000","last_became_peered":"2023-08-25T02:25:04.234588+0000","last_unstale":"2023-08-25T02:25:04.236090+0000","last_undegraded":"2023-08-25T02:25:04.236090+0000","last_fullsized":"2023-08-25T02:25:04.236090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:12:21.426235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214180+0000","last_change":"2023-08-25T02:25:04.227310+0000","last_active":"2023-08-25T02:25:05.214180+0000","last_peered":"2023-08-25T02:25:05.214180+0000","last_clean":"2023-08-25T02:25:05.214180+0000","last_became_active":"2023-08-25T02:25:04.226940+0000","last_became_peered":"2023-08-25T02:25:04.226940+0000","last_unstale":"2023-08-25T02:25:05.214180+0000","last_undegraded":"2023-08-25T02:25:05.214180+0000","last_fullsized":"2023-08-25T02:25:05.214180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:21:22.991356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156816+0000","last_change":"2023-08-25T02:25:04.230835+0000","last_active":"2023-08-25T02:25:06.156816+0000","last_peered":"2023-08-25T02:25:06.156816+0000","last_clean":"2023-08-25T02:25:06.156816+0000","last_became_active":"2023-08-25T02:25:04.230534+0000","last_became_peered":"2023-08-25T02:25:04.230534+0000","last_unstale":"2023-08-25T02:25:06.156816+0000","last_undegraded":"2023-08-25T02:25:06.156816+0000","last_fullsized":"2023-08-25T02:25:06.156816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:09.527955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213887+0000","last_change":"2023-08-25T02:25:04.224250+0000","last_active":"2023-08-25T02:25:05.213887+0000","last_peered":"2023-08-25T02:25:05.213887+0000","last_clean":"2023-08-25T02:25:05.213887+0000","last_became_active":"2023-08-25T02:25:04.223591+0000","last_became_peered":"2023-08-25T02:25:04.223591+0000","last_unstale":"2023-08-25T02:25:05.213887+0000","last_undegraded":"2023-08-25T02:25:05.213887+0000","last_fullsized":"2023-08-25T02:25:05.213887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:26:52.465008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438356+0000","last_change":"2023-08-25T02:25:04.229455+0000","last_active":"2023-08-25T02:25:06.438356+0000","last_peered":"2023-08-25T02:25:06.438356+0000","last_clean":"2023-08-25T02:25:06.438356+0000","last_became_active":"2023-08-25T02:25:04.228114+0000","last_became_peered":"2023-08-25T02:25:04.228114+0000","last_unstale":"2023-08-25T02:25:06.438356+0000","last_undegraded":"2023-08-25T02:25:06.438356+0000","last_fullsized":"2023-08-25T02:25:06.438356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:12:29.745750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.227555+0000","last_change":"2023-08-25T02:25:04.226625+0000","last_active":"2023-08-25T02:25:04.227555+0000","last_peered":"2023-08-25T02:25:04.227555+0000","last_clean":"2023-08-25T02:25:04.227555+0000","last_became_active":"2023-08-25T02:25:04.226253+0000","last_became_peered":"2023-08-25T02:25:04.226253+0000","last_unstale":"2023-08-25T02:25:04.227555+0000","last_undegraded":"2023-08-25T02:25:04.227555+0000","last_fullsized":"2023-08-25T02:25:04.227555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:48:06.916471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.224861+0000","last_change":"2023-08-25T02:25:04.223776+0000","last_active":"2023-08-25T02:25:04.224861+0000","last_peered":"2023-08-25T02:25:04.224861+0000","last_clean":"2023-08-25T02:25:04.224861+0000","last_became_active":"2023-08-25T02:25:04.223597+0000","last_became_peered":"2023-08-25T02:25:04.223597+0000","last_unstale":"2023-08-25T02:25:04.224861+0000","last_undegraded":"2023-08-25T02:25:04.224861+0000","last_fullsized":"2023-08-25T02:25:04.224861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:38:17.316678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3636,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":8546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":831488,"data_stored":393540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":10,"up_from":68,"seq":292057776143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1003361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.37},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.622},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":9,"up_from":63,"seq":270582939668,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":403199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.629}]},{"osd":4,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.50900000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.55000000000000004}]}]},{"osd":8,"up_from":56,"seq":240518168600,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":411813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.745}]},{"osd":7,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":7,"up_from":49,"seq":210453397533,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1454080,"data_stored":1002137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.497}]},{"osd":1,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42599999999999999,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.432},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60199999999999998,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.628}]},{"osd":4,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.77700000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.65200000000000002,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92100000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70499999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.88900000000000001}]},{"osd":8,"last update":"Fri Aug 25 02:24:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.83699999999999997}]},{"osd":9,"last update":"Fri Aug 25 02:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.90900000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.80500000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]},{"osd":6,"up_from":44,"seq":188978561057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":843776,"data_stored":402617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.373},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.38400000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.442,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72699999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46600000000000003,"15min":0.46100000000000002},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.433},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44400000000000001,"15min":0.443},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59999999999999998,"15min":0.59299999999999997},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.53400000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86499999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.46800000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.62},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.56200000000000006}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.752}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":412526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47999999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.51600000000000001,"15min":0.48999999999999999},"min":{"1min":0.44600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.495,"15min":0.48299999999999998},"min":{"1min":0.37,"5min":0.307,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Aug 25 02:24:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.372,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83399999999999996,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.39300000000000002},"min":{"1min":0.33600000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.45900000000000002,"15min":0.41999999999999998},"min":{"1min":0.36499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.42099999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57299999999999995,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.441,"15min":0.441},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.502,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.48799999999999999,"5min":0.434,"15min":0.434},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.621},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.95799999999999996}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.72299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":1.014}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":847872,"data_stored":402707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.434,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Aug 25 02:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.33900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55800000000000005,"15min":0.55000000000000004},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.53700000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.64700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.32400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71699999999999997,"5min":0.754,"15min":0.754},"last":0.47699999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83899999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73399999999999999}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.49299999999999999}]},{"osd":10,"last update":"Fri Aug 25 02:25:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":860160,"data_stored":404263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.499,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.48999999999999999,"15min":0.46200000000000002},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45200000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54600000000000004,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.497},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.53300000000000003},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Aug 25 02:24:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76800000000000002,"5min":0.871,"15min":0.871},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.435,"15min":0.42999999999999999},"min":{"1min":0.33700000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.433,"15min":0.42799999999999999},"min":{"1min":0.32900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.378}]},{"osd":6,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64100000000000001},"min":{"1min":0.47299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51700000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.65600000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57999999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":411291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.312},"min":{"1min":0.25800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.311,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.52500000000000002,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.56799999999999995,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.52600000000000002},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.55600000000000005,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Aug 25 02:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.59299999999999997},"min":{"1min":0.46500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53300000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.48299999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.495}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57999999999999996}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.59099999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":860160,"data_stored":411209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.309,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.38300000000000001,"15min":0.36099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.44700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52600000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.442}]},{"osd":6,"last update":"Fri Aug 25 02:24:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56499999999999995}]},{"osd":8,"last update":"Fri Aug 25 02:24:22 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.51600000000000001}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57999999999999996}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.504},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1002700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.496},"min":{"1min":0.36399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.314}]},{"osd":1,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.27600000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56000000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28299999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72199999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.40999999999999998}]},{"osd":4,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60699999999999998,"15min":0.60499999999999998},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54300000000000004}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.501},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.67600000000000005,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.59199999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.436}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri Aug 25 02:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T02:25:10.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T02:25:10.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- 172.21.15.89:0/2013875682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b01024d0 msgr2=0x7f19b0108d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- 172.21.15.89:0/2013875682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b01024d0 0x7f19b0108d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19a800b0d0 tx=0x7f19a800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- 172.21.15.89:0/2013875682 shutdown_connections 2023-08-25T02:25:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- 172.21.15.89:0/2013875682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 0x7f19b0100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- 172.21.15.89:0/2013875682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b01024d0 0x7f19b0108d80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- 172.21.15.89:0/2013875682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0101b30 0x7f19b0101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- 172.21.15.89:0/2013875682 >> 172.21.15.89:0/2013875682 conn(0x7f19b00fbe90 msgr2=0x7f19b00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- 172.21.15.89:0/2013875682 shutdown_connections 2023-08-25T02:25:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- 172.21.15.89:0/2013875682 wait complete. 2023-08-25T02:25:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 Processor -- start 2023-08-25T02:25:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- start start 2023-08-25T02:25:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 0x7f19b01988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b01024d0 0x7f19b019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01a0470 con 0x7f19b0101b30 2023-08-25T02:25:10.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01a05b0 con 0x7f19b0100940 2023-08-25T02:25:10.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.883+0000 7f19b785c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01a0740 con 0x7f19b01024d0 2023-08-25T02:25:10.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46806/0 (socket says 172.21.15.89:46806) 2023-08-25T02:25:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 -- 172.21.15.89:0/53086545 learned_addr learned my addr 172.21.15.89:0/53086545 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b5df9700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b01024d0 0x7f19b019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 -- 172.21.15.89:0/53086545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b01024d0 msgr2=0x7f19b019fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b01024d0 0x7f19b019fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 -- 172.21.15.89:0/53086545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 msgr2=0x7f19b01988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:10.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b55f8700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 0x7f19b01988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 0x7f19b01988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:10.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 -- 172.21.15.89:0/53086545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a8009d00 con 0x7f19b0101b30 2023-08-25T02:25:10.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b4df7700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e9a0 tx=0x7f19a000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0009790 con 0x7f19b0101b30 2023-08-25T02:25:10.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19a000cbd0 con 0x7f19b0101b30 2023-08-25T02:25:10.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0018680 con 0x7f19b0101b30 2023-08-25T02:25:10.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b01a0a20 con 0x7f19b0101b30 2023-08-25T02:25:10.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.884+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b01a0f60 con 0x7f19b0101b30 2023-08-25T02:25:10.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.885+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b010c1f0 con 0x7f19b0101b30 2023-08-25T02:25:10.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.886+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f19a000cd40 con 0x7f19b0101b30 2023-08-25T02:25:10.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.886+0000 7f19a67fc700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 0x7f199c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:10.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.888+0000 7f19b55f8700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 0x7f199c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:10.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.888+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f19a0014070 con 0x7f19b0101b30 2023-08-25T02:25:10.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.888+0000 7f19b55f8700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 0x7f199c0a2460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19ac006fd0 tx=0x7f19ac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:10.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:10.892+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f19a0051e00 con 0x7f19b0101b30 2023-08-25T02:25:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: pgmap v253: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-08-25T02:25:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:active 2023-08-25T02:25:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-08-25T02:25:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-08-25T02:25:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:25:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4032376897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/221879686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: pgmap v253: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-08-25T02:25:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:active 2023-08-25T02:25:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-08-25T02:25:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-08-25T02:25:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:25:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4032376897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/221879686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:11.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.076+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19b0108390 con 0x7f199c09ffa0 2023-08-25T02:25:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: pgmap v253: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 10 op/s 2023-08-25T02:25:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] up:active 2023-08-25T02:25:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2725725203' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-08-25T02:25:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-08-25T02:25:11.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:25:11.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4032376897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:11.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/221879686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.086+0000 7f19a67fc700 1 -- 172.21.15.89:0/53086545 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335584 (secure 0 0 0) 0x7f19b0108390 con 0x7f199c09ffa0 2023-08-25T02:25:11.090 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 msgr2=0x7f199c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 0x7f199c0a2460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19ac006fd0 tx=0x7f19ac008040 comp rx=0 tx=0).stop 2023-08-25T02:25:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 msgr2=0x7f19b0198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e9a0 tx=0x7f19a000ed60 comp rx=0 tx=0).stop 2023-08-25T02:25:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 shutdown_connections 2023-08-25T02:25:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f199c09ffa0 0x7f199c0a2460 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b0100940 0x7f19b01988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b01024d0 0x7f19b019fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 --2- 172.21.15.89:0/53086545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b0101b30 0x7f19b0198e30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.089+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 >> 172.21.15.89:0/53086545 conn(0x7f19b00fbe90 msgr2=0x7f19b00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.090+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 shutdown_connections 2023-08-25T02:25:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:11.090+0000 7f19b785c700 1 -- 172.21.15.89:0/53086545 wait complete. 2023-08-25T02:25:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-08-25T02:25:11.103 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2023-08-25T02:25:09.296350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594004,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":244,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336056,"kb_used_data":11808,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605768,"statfs":{"total":1151940427776,"available":1151596306432,"internally_reserved":0,"allocated":12091392,"data_stored":6661363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3636,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001210"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.227164+0000","last_change":"2023-08-25T02:25:04.226503+0000","last_active":"2023-08-25T02:25:04.227164+0000","last_peered":"2023-08-25T02:25:04.227164+0000","last_clean":"2023-08-25T02:25:04.227164+0000","last_became_active":"2023-08-25T02:25:04.226184+0000","last_became_peered":"2023-08-25T02:25:04.226184+0000","last_unstale":"2023-08-25T02:25:04.227164+0000","last_undegraded":"2023-08-25T02:25:04.227164+0000","last_fullsized":"2023-08-25T02:25:04.227164+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:12:59.270865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938361+0000","last_change":"2023-08-25T02:25:04.236479+0000","last_active":"2023-08-25T02:25:05.938361+0000","last_peered":"2023-08-25T02:25:05.938361+0000","last_clean":"2023-08-25T02:25:05.938361+0000","last_became_active":"2023-08-25T02:25:04.235376+0000","last_became_peered":"2023-08-25T02:25:04.235376+0000","last_unstale":"2023-08-25T02:25:05.938361+0000","last_undegraded":"2023-08-25T02:25:05.938361+0000","last_fullsized":"2023-08-25T02:25:05.938361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:47.418998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938362+0000","last_change":"2023-08-25T02:25:04.235840+0000","last_active":"2023-08-25T02:25:05.938362+0000","last_peered":"2023-08-25T02:25:05.938362+0000","last_clean":"2023-08-25T02:25:05.938362+0000","last_became_active":"2023-08-25T02:25:04.235279+0000","last_became_peered":"2023-08-25T02:25:04.235279+0000","last_unstale":"2023-08-25T02:25:05.938362+0000","last_undegraded":"2023-08-25T02:25:05.938362+0000","last_fullsized":"2023-08-25T02:25:05.938362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:23.103019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474317+0000","last_change":"2023-08-25T02:25:04.237793+0000","last_active":"2023-08-25T02:25:06.474317+0000","last_peered":"2023-08-25T02:25:06.474317+0000","last_clean":"2023-08-25T02:25:06.474317+0000","last_became_active":"2023-08-25T02:25:04.237493+0000","last_became_peered":"2023-08-25T02:25:04.237493+0000","last_unstale":"2023-08-25T02:25:06.474317+0000","last_undegraded":"2023-08-25T02:25:06.474317+0000","last_fullsized":"2023-08-25T02:25:06.474317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:36:34.299231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657466+0000","last_change":"2023-08-25T02:25:04.221053+0000","last_active":"2023-08-25T02:25:06.657466+0000","last_peered":"2023-08-25T02:25:06.657466+0000","last_clean":"2023-08-25T02:25:06.657466+0000","last_became_active":"2023-08-25T02:25:04.220710+0000","last_became_peered":"2023-08-25T02:25:04.220710+0000","last_unstale":"2023-08-25T02:25:06.657466+0000","last_undegraded":"2023-08-25T02:25:06.657466+0000","last_fullsized":"2023-08-25T02:25:06.657466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:25:14.107245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214409+0000","last_change":"2023-08-25T02:25:04.234380+0000","last_active":"2023-08-25T02:25:05.214409+0000","last_peered":"2023-08-25T02:25:05.214409+0000","last_clean":"2023-08-25T02:25:05.214409+0000","last_became_active":"2023-08-25T02:25:04.233376+0000","last_became_peered":"2023-08-25T02:25:04.233376+0000","last_unstale":"2023-08-25T02:25:05.214409+0000","last_undegraded":"2023-08-25T02:25:05.214409+0000","last_fullsized":"2023-08-25T02:25:05.214409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:18:49.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473517+0000","last_change":"2023-08-25T02:25:04.228398+0000","last_active":"2023-08-25T02:25:06.473517+0000","last_peered":"2023-08-25T02:25:06.473517+0000","last_clean":"2023-08-25T02:25:06.473517+0000","last_became_active":"2023-08-25T02:25:04.227310+0000","last_became_peered":"2023-08-25T02:25:04.227310+0000","last_unstale":"2023-08-25T02:25:06.473517+0000","last_undegraded":"2023-08-25T02:25:06.473517+0000","last_fullsized":"2023-08-25T02:25:06.473517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:45:46.502139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213776+0000","last_change":"2023-08-25T02:25:04.235175+0000","last_active":"2023-08-25T02:25:05.213776+0000","last_peered":"2023-08-25T02:25:05.213776+0000","last_clean":"2023-08-25T02:25:05.213776+0000","last_became_active":"2023-08-25T02:25:04.234048+0000","last_became_peered":"2023-08-25T02:25:04.234048+0000","last_unstale":"2023-08-25T02:25:05.213776+0000","last_undegraded":"2023-08-25T02:25:05.213776+0000","last_fullsized":"2023-08-25T02:25:05.213776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:20.347389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438643+0000","last_change":"2023-08-25T02:25:04.226475+0000","last_active":"2023-08-25T02:25:06.438643+0000","last_peered":"2023-08-25T02:25:06.438643+0000","last_clean":"2023-08-25T02:25:06.438643+0000","last_became_active":"2023-08-25T02:25:04.226125+0000","last_became_peered":"2023-08-25T02:25:04.226125+0000","last_unstale":"2023-08-25T02:25:06.438643+0000","last_undegraded":"2023-08-25T02:25:06.438643+0000","last_fullsized":"2023-08-25T02:25:06.438643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:44.888916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214553+0000","last_change":"2023-08-25T02:25:04.234476+0000","last_active":"2023-08-25T02:25:05.214553+0000","last_peered":"2023-08-25T02:25:05.214553+0000","last_clean":"2023-08-25T02:25:05.214553+0000","last_became_active":"2023-08-25T02:25:04.234062+0000","last_became_peered":"2023-08-25T02:25:04.234062+0000","last_unstale":"2023-08-25T02:25:05.214553+0000","last_undegraded":"2023-08-25T02:25:05.214553+0000","last_fullsized":"2023-08-25T02:25:05.214553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:35:10.280467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473970+0000","last_change":"2023-08-25T02:25:04.233218+0000","last_active":"2023-08-25T02:25:06.473970+0000","last_peered":"2023-08-25T02:25:06.473970+0000","last_clean":"2023-08-25T02:25:06.473970+0000","last_became_active":"2023-08-25T02:25:04.232933+0000","last_became_peered":"2023-08-25T02:25:04.232933+0000","last_unstale":"2023-08-25T02:25:06.473970+0000","last_undegraded":"2023-08-25T02:25:06.473970+0000","last_fullsized":"2023-08-25T02:25:06.473970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:32:06.548589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474291+0000","last_change":"2023-08-25T02:25:04.238646+0000","last_active":"2023-08-25T02:25:06.474291+0000","last_peered":"2023-08-25T02:25:06.474291+0000","last_clean":"2023-08-25T02:25:06.474291+0000","last_became_active":"2023-08-25T02:25:04.238252+0000","last_became_peered":"2023-08-25T02:25:04.238252+0000","last_unstale":"2023-08-25T02:25:06.474291+0000","last_undegraded":"2023-08-25T02:25:06.474291+0000","last_fullsized":"2023-08-25T02:25:06.474291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:43:32.951632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236750+0000","last_change":"2023-08-25T02:25:04.236025+0000","last_active":"2023-08-25T02:25:04.236750+0000","last_peered":"2023-08-25T02:25:04.236750+0000","last_clean":"2023-08-25T02:25:04.236750+0000","last_became_active":"2023-08-25T02:25:04.235682+0000","last_became_peered":"2023-08-25T02:25:04.235682+0000","last_unstale":"2023-08-25T02:25:04.236750+0000","last_undegraded":"2023-08-25T02:25:04.236750+0000","last_fullsized":"2023-08-25T02:25:04.236750+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:15.209256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938102+0000","last_change":"2023-08-25T02:25:04.236295+0000","last_active":"2023-08-25T02:25:05.938102+0000","last_peered":"2023-08-25T02:25:05.938102+0000","last_clean":"2023-08-25T02:25:05.938102+0000","last_became_active":"2023-08-25T02:25:04.235979+0000","last_became_peered":"2023-08-25T02:25:04.235979+0000","last_unstale":"2023-08-25T02:25:05.938102+0000","last_undegraded":"2023-08-25T02:25:05.938102+0000","last_fullsized":"2023-08-25T02:25:05.938102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:12:13.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938095+0000","last_change":"2023-08-25T02:25:04.234885+0000","last_active":"2023-08-25T02:25:05.938095+0000","last_peered":"2023-08-25T02:25:05.938095+0000","last_clean":"2023-08-25T02:25:05.938095+0000","last_became_active":"2023-08-25T02:25:04.234603+0000","last_became_peered":"2023-08-25T02:25:04.234603+0000","last_unstale":"2023-08-25T02:25:05.938095+0000","last_undegraded":"2023-08-25T02:25:05.938095+0000","last_fullsized":"2023-08-25T02:25:05.938095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:26.095674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438191+0000","last_change":"2023-08-25T02:25:04.230151+0000","last_active":"2023-08-25T02:25:06.438191+0000","last_peered":"2023-08-25T02:25:06.438191+0000","last_clean":"2023-08-25T02:25:06.438191+0000","last_became_active":"2023-08-25T02:25:04.227881+0000","last_became_peered":"2023-08-25T02:25:04.227881+0000","last_unstale":"2023-08-25T02:25:06.438191+0000","last_undegraded":"2023-08-25T02:25:06.438191+0000","last_fullsized":"2023-08-25T02:25:06.438191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:31.382761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236507+0000","last_change":"2023-08-25T02:25:04.230117+0000","last_active":"2023-08-25T02:25:04.236507+0000","last_peered":"2023-08-25T02:25:04.236507+0000","last_clean":"2023-08-25T02:25:04.236507+0000","last_became_active":"2023-08-25T02:25:04.229810+0000","last_became_peered":"2023-08-25T02:25:04.229810+0000","last_unstale":"2023-08-25T02:25:04.236507+0000","last_undegraded":"2023-08-25T02:25:04.236507+0000","last_fullsized":"2023-08-25T02:25:04.236507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:25:31.424239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.235629+0000","last_change":"2023-08-25T02:25:04.235011+0000","last_active":"2023-08-25T02:25:04.235629+0000","last_peered":"2023-08-25T02:25:04.235629+0000","last_clean":"2023-08-25T02:25:04.235629+0000","last_became_active":"2023-08-25T02:25:04.234664+0000","last_became_peered":"2023-08-25T02:25:04.234664+0000","last_unstale":"2023-08-25T02:25:04.235629+0000","last_undegraded":"2023-08-25T02:25:04.235629+0000","last_fullsized":"2023-08-25T02:25:04.235629+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:22:16.915760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230565+0000","last_change":"2023-08-25T02:25:04.229223+0000","last_active":"2023-08-25T02:25:04.230565+0000","last_peered":"2023-08-25T02:25:04.230565+0000","last_clean":"2023-08-25T02:25:04.230565+0000","last_became_active":"2023-08-25T02:25:04.227656+0000","last_became_peered":"2023-08-25T02:25:04.227656+0000","last_unstale":"2023-08-25T02:25:04.230565+0000","last_undegraded":"2023-08-25T02:25:04.230565+0000","last_fullsized":"2023-08-25T02:25:04.230565+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:46:08.334390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.611544+0000","last_change":"2023-08-25T02:25:03.613357+0000","last_active":"2023-08-25T02:25:04.611544+0000","last_peered":"2023-08-25T02:25:04.611544+0000","last_clean":"2023-08-25T02:25:04.611544+0000","last_became_active":"2023-08-25T02:25:03.612977+0000","last_became_peered":"2023-08-25T02:25:03.612977+0000","last_unstale":"2023-08-25T02:25:04.611544+0000","last_undegraded":"2023-08-25T02:25:04.611544+0000","last_fullsized":"2023-08-25T02:25:04.611544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:14:21.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214090+0000","last_change":"2023-08-25T02:25:04.230544+0000","last_active":"2023-08-25T02:25:05.214090+0000","last_peered":"2023-08-25T02:25:05.214090+0000","last_clean":"2023-08-25T02:25:05.214090+0000","last_became_active":"2023-08-25T02:25:04.230226+0000","last_became_peered":"2023-08-25T02:25:04.230226+0000","last_unstale":"2023-08-25T02:25:05.214090+0000","last_undegraded":"2023-08-25T02:25:05.214090+0000","last_fullsized":"2023-08-25T02:25:05.214090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:41.647707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214191+0000","last_change":"2023-08-25T02:25:03.617626+0000","last_active":"2023-08-25T02:25:05.214191+0000","last_peered":"2023-08-25T02:25:05.214191+0000","last_clean":"2023-08-25T02:25:05.214191+0000","last_became_active":"2023-08-25T02:25:03.617323+0000","last_became_peered":"2023-08-25T02:25:03.617323+0000","last_unstale":"2023-08-25T02:25:05.214191+0000","last_undegraded":"2023-08-25T02:25:05.214191+0000","last_fullsized":"2023-08-25T02:25:05.214191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:05.360379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938528+0000","last_change":"2023-08-25T02:25:04.226808+0000","last_active":"2023-08-25T02:25:05.938528+0000","last_peered":"2023-08-25T02:25:05.938528+0000","last_clean":"2023-08-25T02:25:05.938528+0000","last_became_active":"2023-08-25T02:25:04.226407+0000","last_became_peered":"2023-08-25T02:25:04.226407+0000","last_unstale":"2023-08-25T02:25:05.938528+0000","last_undegraded":"2023-08-25T02:25:05.938528+0000","last_fullsized":"2023-08-25T02:25:05.938528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:26:23.478853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657716+0000","last_change":"2023-08-25T02:25:03.611831+0000","last_active":"2023-08-25T02:25:06.657716+0000","last_peered":"2023-08-25T02:25:06.657716+0000","last_clean":"2023-08-25T02:25:06.657716+0000","last_became_active":"2023-08-25T02:25:03.611530+0000","last_became_peered":"2023-08-25T02:25:03.611530+0000","last_unstale":"2023-08-25T02:25:06.657716+0000","last_undegraded":"2023-08-25T02:25:06.657716+0000","last_fullsized":"2023-08-25T02:25:06.657716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:02:49.263476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438337+0000","last_change":"2023-08-25T02:25:04.236572+0000","last_active":"2023-08-25T02:25:06.438337+0000","last_peered":"2023-08-25T02:25:06.438337+0000","last_clean":"2023-08-25T02:25:06.438337+0000","last_became_active":"2023-08-25T02:25:04.236242+0000","last_became_peered":"2023-08-25T02:25:04.236242+0000","last_unstale":"2023-08-25T02:25:06.438337+0000","last_undegraded":"2023-08-25T02:25:06.438337+0000","last_fullsized":"2023-08-25T02:25:06.438337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:45:50.267973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657579+0000","last_change":"2023-08-25T02:25:03.616265+0000","last_active":"2023-08-25T02:25:06.657579+0000","last_peered":"2023-08-25T02:25:06.657579+0000","last_clean":"2023-08-25T02:25:06.657579+0000","last_became_active":"2023-08-25T02:25:03.615926+0000","last_became_peered":"2023-08-25T02:25:03.615926+0000","last_unstale":"2023-08-25T02:25:06.657579+0000","last_undegraded":"2023-08-25T02:25:06.657579+0000","last_fullsized":"2023-08-25T02:25:06.657579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:21:34.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157129+0000","last_change":"2023-08-25T02:25:04.232925+0000","last_active":"2023-08-25T02:25:06.157129+0000","last_peered":"2023-08-25T02:25:06.157129+0000","last_clean":"2023-08-25T02:25:06.157129+0000","last_became_active":"2023-08-25T02:25:04.232615+0000","last_became_peered":"2023-08-25T02:25:04.232615+0000","last_unstale":"2023-08-25T02:25:06.157129+0000","last_undegraded":"2023-08-25T02:25:06.157129+0000","last_fullsized":"2023-08-25T02:25:06.157129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:48:54.025387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633789+0000","last_change":"2023-08-25T02:25:03.618534+0000","last_active":"2023-08-25T02:25:07.633789+0000","last_peered":"2023-08-25T02:25:07.633789+0000","last_clean":"2023-08-25T02:25:07.633789+0000","last_became_active":"2023-08-25T02:25:03.618284+0000","last_became_peered":"2023-08-25T02:25:03.618284+0000","last_unstale":"2023-08-25T02:25:07.633789+0000","last_undegraded":"2023-08-25T02:25:07.633789+0000","last_fullsized":"2023-08-25T02:25:07.633789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:55:34.354817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657492+0000","last_change":"2023-08-25T02:25:04.225148+0000","last_active":"2023-08-25T02:25:06.657492+0000","last_peered":"2023-08-25T02:25:06.657492+0000","last_clean":"2023-08-25T02:25:06.657492+0000","last_became_active":"2023-08-25T02:25:04.224776+0000","last_became_peered":"2023-08-25T02:25:04.224776+0000","last_unstale":"2023-08-25T02:25:06.657492+0000","last_undegraded":"2023-08-25T02:25:06.657492+0000","last_fullsized":"2023-08-25T02:25:06.657492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:27.220482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214314+0000","last_change":"2023-08-25T02:25:03.610869+0000","last_active":"2023-08-25T02:25:05.214314+0000","last_peered":"2023-08-25T02:25:05.214314+0000","last_clean":"2023-08-25T02:25:05.214314+0000","last_became_active":"2023-08-25T02:25:03.610568+0000","last_became_peered":"2023-08-25T02:25:03.610568+0000","last_unstale":"2023-08-25T02:25:05.214314+0000","last_undegraded":"2023-08-25T02:25:05.214314+0000","last_fullsized":"2023-08-25T02:25:05.214314+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:16.686802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157177+0000","last_change":"2023-08-25T02:25:04.230241+0000","last_active":"2023-08-25T02:25:06.157177+0000","last_peered":"2023-08-25T02:25:06.157177+0000","last_clean":"2023-08-25T02:25:06.157177+0000","last_became_active":"2023-08-25T02:25:04.229871+0000","last_became_peered":"2023-08-25T02:25:04.229871+0000","last_unstale":"2023-08-25T02:25:06.157177+0000","last_undegraded":"2023-08-25T02:25:06.157177+0000","last_fullsized":"2023-08-25T02:25:06.157177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:20:16.764817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.217198+0000","last_change":"2023-08-25T02:25:03.610650+0000","last_active":"2023-08-25T02:25:04.217198+0000","last_peered":"2023-08-25T02:25:04.217198+0000","last_clean":"2023-08-25T02:25:04.217198+0000","last_became_active":"2023-08-25T02:25:03.609901+0000","last_became_peered":"2023-08-25T02:25:03.609901+0000","last_unstale":"2023-08-25T02:25:04.217198+0000","last_undegraded":"2023-08-25T02:25:04.217198+0000","last_fullsized":"2023-08-25T02:25:04.217198+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:58.744550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213759+0000","last_change":"2023-08-25T02:25:04.222682+0000","last_active":"2023-08-25T02:25:05.213759+0000","last_peered":"2023-08-25T02:25:05.213759+0000","last_clean":"2023-08-25T02:25:05.213759+0000","last_became_active":"2023-08-25T02:25:04.221791+0000","last_became_peered":"2023-08-25T02:25:04.221791+0000","last_unstale":"2023-08-25T02:25:05.213759+0000","last_undegraded":"2023-08-25T02:25:05.213759+0000","last_fullsized":"2023-08-25T02:25:05.213759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:29:20.943592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156809+0000","last_change":"2023-08-25T02:25:03.610342+0000","last_active":"2023-08-25T02:25:06.156809+0000","last_peered":"2023-08-25T02:25:06.156809+0000","last_clean":"2023-08-25T02:25:06.156809+0000","last_became_active":"2023-08-25T02:25:03.609928+0000","last_became_peered":"2023-08-25T02:25:03.609928+0000","last_unstale":"2023-08-25T02:25:06.156809+0000","last_undegraded":"2023-08-25T02:25:06.156809+0000","last_fullsized":"2023-08-25T02:25:06.156809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:42.862108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.235064+0000","last_change":"2023-08-25T02:25:04.234364+0000","last_active":"2023-08-25T02:25:04.235064+0000","last_peered":"2023-08-25T02:25:04.235064+0000","last_clean":"2023-08-25T02:25:04.235064+0000","last_became_active":"2023-08-25T02:25:04.234050+0000","last_became_peered":"2023-08-25T02:25:04.234050+0000","last_unstale":"2023-08-25T02:25:04.235064+0000","last_undegraded":"2023-08-25T02:25:04.235064+0000","last_fullsized":"2023-08-25T02:25:04.235064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:18:36.177255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214477+0000","last_change":"2023-08-25T02:25:03.617016+0000","last_active":"2023-08-25T02:25:05.214477+0000","last_peered":"2023-08-25T02:25:05.214477+0000","last_clean":"2023-08-25T02:25:05.214477+0000","last_became_active":"2023-08-25T02:25:03.616670+0000","last_became_peered":"2023-08-25T02:25:03.616670+0000","last_unstale":"2023-08-25T02:25:05.214477+0000","last_undegraded":"2023-08-25T02:25:05.214477+0000","last_fullsized":"2023-08-25T02:25:05.214477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:42:03.302702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438625+0000","last_change":"2023-08-25T02:25:04.236009+0000","last_active":"2023-08-25T02:25:06.438625+0000","last_peered":"2023-08-25T02:25:06.438625+0000","last_clean":"2023-08-25T02:25:06.438625+0000","last_became_active":"2023-08-25T02:25:04.235722+0000","last_became_peered":"2023-08-25T02:25:04.235722+0000","last_unstale":"2023-08-25T02:25:06.438625+0000","last_undegraded":"2023-08-25T02:25:06.438625+0000","last_fullsized":"2023-08-25T02:25:06.438625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:42:44.656781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474018+0000","last_change":"2023-08-25T02:25:03.611060+0000","last_active":"2023-08-25T02:25:06.474018+0000","last_peered":"2023-08-25T02:25:06.474018+0000","last_clean":"2023-08-25T02:25:06.474018+0000","last_became_active":"2023-08-25T02:25:03.610715+0000","last_became_peered":"2023-08-25T02:25:03.610715+0000","last_unstale":"2023-08-25T02:25:06.474018+0000","last_undegraded":"2023-08-25T02:25:06.474018+0000","last_fullsized":"2023-08-25T02:25:06.474018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:17.684716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938519+0000","last_change":"2023-08-25T02:25:04.225855+0000","last_active":"2023-08-25T02:25:05.938519+0000","last_peered":"2023-08-25T02:25:05.938519+0000","last_clean":"2023-08-25T02:25:05.938519+0000","last_became_active":"2023-08-25T02:25:04.225559+0000","last_became_peered":"2023-08-25T02:25:04.225559+0000","last_unstale":"2023-08-25T02:25:05.938519+0000","last_undegraded":"2023-08-25T02:25:05.938519+0000","last_fullsized":"2023-08-25T02:25:05.938519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:33:19.054268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157230+0000","last_change":"2023-08-25T02:25:03.617097+0000","last_active":"2023-08-25T02:25:06.157230+0000","last_peered":"2023-08-25T02:25:06.157230+0000","last_clean":"2023-08-25T02:25:06.157230+0000","last_became_active":"2023-08-25T02:25:03.616809+0000","last_became_peered":"2023-08-25T02:25:03.616809+0000","last_unstale":"2023-08-25T02:25:06.157230+0000","last_undegraded":"2023-08-25T02:25:06.157230+0000","last_fullsized":"2023-08-25T02:25:06.157230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:11:24.908899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438483+0000","last_change":"2023-08-25T02:25:04.226384+0000","last_active":"2023-08-25T02:25:06.438483+0000","last_peered":"2023-08-25T02:25:06.438483+0000","last_clean":"2023-08-25T02:25:06.438483+0000","last_became_active":"2023-08-25T02:25:04.226047+0000","last_became_peered":"2023-08-25T02:25:04.226047+0000","last_unstale":"2023-08-25T02:25:06.438483+0000","last_undegraded":"2023-08-25T02:25:06.438483+0000","last_fullsized":"2023-08-25T02:25:06.438483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:01.485924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214517+0000","last_change":"2023-08-25T02:25:03.609570+0000","last_active":"2023-08-25T02:25:05.214517+0000","last_peered":"2023-08-25T02:25:05.214517+0000","last_clean":"2023-08-25T02:25:05.214517+0000","last_became_active":"2023-08-25T02:25:03.609253+0000","last_became_peered":"2023-08-25T02:25:03.609253+0000","last_unstale":"2023-08-25T02:25:05.214517+0000","last_undegraded":"2023-08-25T02:25:05.214517+0000","last_fullsized":"2023-08-25T02:25:05.214517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:14.775308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657588+0000","last_change":"2023-08-25T02:25:04.226667+0000","last_active":"2023-08-25T02:25:06.657588+0000","last_peered":"2023-08-25T02:25:06.657588+0000","last_clean":"2023-08-25T02:25:06.657588+0000","last_became_active":"2023-08-25T02:25:04.226332+0000","last_became_peered":"2023-08-25T02:25:04.226332+0000","last_unstale":"2023-08-25T02:25:06.657588+0000","last_undegraded":"2023-08-25T02:25:06.657588+0000","last_fullsized":"2023-08-25T02:25:06.657588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:51:57.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.610718+0000","last_change":"2023-08-25T02:25:03.609419+0000","last_active":"2023-08-25T02:25:04.610718+0000","last_peered":"2023-08-25T02:25:04.610718+0000","last_clean":"2023-08-25T02:25:04.610718+0000","last_became_active":"2023-08-25T02:25:03.609101+0000","last_became_peered":"2023-08-25T02:25:03.609101+0000","last_unstale":"2023-08-25T02:25:04.610718+0000","last_undegraded":"2023-08-25T02:25:04.610718+0000","last_fullsized":"2023-08-25T02:25:04.610718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:54:48.243106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232161+0000","last_change":"2023-08-25T02:25:04.231351+0000","last_active":"2023-08-25T02:25:04.232161+0000","last_peered":"2023-08-25T02:25:04.232161+0000","last_clean":"2023-08-25T02:25:04.232161+0000","last_became_active":"2023-08-25T02:25:04.230974+0000","last_became_peered":"2023-08-25T02:25:04.230974+0000","last_unstale":"2023-08-25T02:25:04.232161+0000","last_undegraded":"2023-08-25T02:25:04.232161+0000","last_fullsized":"2023-08-25T02:25:04.232161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:55:37.390357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633217+0000","last_change":"2023-08-25T02:25:03.617889+0000","last_active":"2023-08-25T02:25:07.633217+0000","last_peered":"2023-08-25T02:25:07.633217+0000","last_clean":"2023-08-25T02:25:07.633217+0000","last_became_active":"2023-08-25T02:25:03.617576+0000","last_became_peered":"2023-08-25T02:25:03.617576+0000","last_unstale":"2023-08-25T02:25:07.633217+0000","last_undegraded":"2023-08-25T02:25:07.633217+0000","last_fullsized":"2023-08-25T02:25:07.633217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:38:22.627438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473924+0000","last_change":"2023-08-25T02:25:04.230135+0000","last_active":"2023-08-25T02:25:06.473924+0000","last_peered":"2023-08-25T02:25:06.473924+0000","last_clean":"2023-08-25T02:25:06.473924+0000","last_became_active":"2023-08-25T02:25:04.229592+0000","last_became_peered":"2023-08-25T02:25:04.229592+0000","last_unstale":"2023-08-25T02:25:06.473924+0000","last_undegraded":"2023-08-25T02:25:06.473924+0000","last_fullsized":"2023-08-25T02:25:06.473924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:28:47.602421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473908+0000","last_change":"2023-08-25T02:25:03.610021+0000","last_active":"2023-08-25T02:25:06.473908+0000","last_peered":"2023-08-25T02:25:06.473908+0000","last_clean":"2023-08-25T02:25:06.473908+0000","last_became_active":"2023-08-25T02:25:03.609703+0000","last_became_peered":"2023-08-25T02:25:03.609703+0000","last_unstale":"2023-08-25T02:25:06.473908+0000","last_undegraded":"2023-08-25T02:25:06.473908+0000","last_fullsized":"2023-08-25T02:25:06.473908+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:25.169266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473493+0000","last_change":"2023-08-25T02:25:04.234643+0000","last_active":"2023-08-25T02:25:06.473493+0000","last_peered":"2023-08-25T02:25:06.473493+0000","last_clean":"2023-08-25T02:25:06.473493+0000","last_became_active":"2023-08-25T02:25:04.234349+0000","last_became_peered":"2023-08-25T02:25:04.234349+0000","last_unstale":"2023-08-25T02:25:06.473493+0000","last_undegraded":"2023-08-25T02:25:06.473493+0000","last_fullsized":"2023-08-25T02:25:06.473493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:08:25.591726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'10","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632840+0000","last_change":"2023-08-25T02:25:03.613702+0000","last_active":"2023-08-25T02:25:07.632840+0000","last_peered":"2023-08-25T02:25:07.632840+0000","last_clean":"2023-08-25T02:25:07.632840+0000","last_became_active":"2023-08-25T02:25:03.613383+0000","last_became_peered":"2023-08-25T02:25:03.613383+0000","last_unstale":"2023-08-25T02:25:07.632840+0000","last_undegraded":"2023-08-25T02:25:07.632840+0000","last_fullsized":"2023-08-25T02:25:07.632840+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:53.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156815+0000","last_change":"2023-08-25T02:25:04.227650+0000","last_active":"2023-08-25T02:25:06.156815+0000","last_peered":"2023-08-25T02:25:06.156815+0000","last_clean":"2023-08-25T02:25:06.156815+0000","last_became_active":"2023-08-25T02:25:04.227335+0000","last_became_peered":"2023-08-25T02:25:04.227335+0000","last_unstale":"2023-08-25T02:25:06.156815+0000","last_undegraded":"2023-08-25T02:25:06.156815+0000","last_fullsized":"2023-08-25T02:25:06.156815+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:10.086931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657623+0000","last_change":"2023-08-25T02:25:03.616065+0000","last_active":"2023-08-25T02:25:06.657623+0000","last_peered":"2023-08-25T02:25:06.657623+0000","last_clean":"2023-08-25T02:25:06.657623+0000","last_became_active":"2023-08-25T02:25:03.615751+0000","last_became_peered":"2023-08-25T02:25:03.615751+0000","last_unstale":"2023-08-25T02:25:06.657623+0000","last_undegraded":"2023-08-25T02:25:06.657623+0000","last_fullsized":"2023-08-25T02:25:06.657623+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:27:27.917233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230930+0000","last_change":"2023-08-25T02:25:04.228285+0000","last_active":"2023-08-25T02:25:04.230930+0000","last_peered":"2023-08-25T02:25:04.230930+0000","last_clean":"2023-08-25T02:25:04.230930+0000","last_became_active":"2023-08-25T02:25:04.228061+0000","last_became_peered":"2023-08-25T02:25:04.228061+0000","last_unstale":"2023-08-25T02:25:04.230930+0000","last_undegraded":"2023-08-25T02:25:04.230930+0000","last_fullsized":"2023-08-25T02:25:04.230930+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:41:57.051512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.218918+0000","last_change":"2023-08-25T02:25:03.615712+0000","last_active":"2023-08-25T02:25:04.218918+0000","last_peered":"2023-08-25T02:25:04.218918+0000","last_clean":"2023-08-25T02:25:04.218918+0000","last_became_active":"2023-08-25T02:25:03.615412+0000","last_became_peered":"2023-08-25T02:25:03.615412+0000","last_unstale":"2023-08-25T02:25:04.218918+0000","last_undegraded":"2023-08-25T02:25:04.218918+0000","last_fullsized":"2023-08-25T02:25:04.218918+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:15.238634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214408+0000","last_change":"2023-08-25T02:25:04.233485+0000","last_active":"2023-08-25T02:25:05.214408+0000","last_peered":"2023-08-25T02:25:05.214408+0000","last_clean":"2023-08-25T02:25:05.214408+0000","last_became_active":"2023-08-25T02:25:04.233205+0000","last_became_peered":"2023-08-25T02:25:04.233205+0000","last_unstale":"2023-08-25T02:25:05.214408+0000","last_undegraded":"2023-08-25T02:25:05.214408+0000","last_fullsized":"2023-08-25T02:25:05.214408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:12.219109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473945+0000","last_change":"2023-08-25T02:25:03.613848+0000","last_active":"2023-08-25T02:25:06.473945+0000","last_peered":"2023-08-25T02:25:06.473945+0000","last_clean":"2023-08-25T02:25:06.473945+0000","last_became_active":"2023-08-25T02:25:03.613538+0000","last_became_peered":"2023-08-25T02:25:03.613538+0000","last_unstale":"2023-08-25T02:25:06.473945+0000","last_undegraded":"2023-08-25T02:25:06.473945+0000","last_fullsized":"2023-08-25T02:25:06.473945+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:52.883965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214475+0000","last_change":"2023-08-25T02:25:04.234167+0000","last_active":"2023-08-25T02:25:05.214475+0000","last_peered":"2023-08-25T02:25:05.214475+0000","last_clean":"2023-08-25T02:25:05.214475+0000","last_became_active":"2023-08-25T02:25:04.233779+0000","last_became_peered":"2023-08-25T02:25:04.233779+0000","last_unstale":"2023-08-25T02:25:05.214475+0000","last_undegraded":"2023-08-25T02:25:05.214475+0000","last_fullsized":"2023-08-25T02:25:05.214475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:19.007690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156909+0000","last_change":"2023-08-25T02:25:04.227759+0000","last_active":"2023-08-25T02:25:06.156909+0000","last_peered":"2023-08-25T02:25:06.156909+0000","last_clean":"2023-08-25T02:25:06.156909+0000","last_became_active":"2023-08-25T02:25:04.227166+0000","last_became_peered":"2023-08-25T02:25:04.227166+0000","last_unstale":"2023-08-25T02:25:06.156909+0000","last_undegraded":"2023-08-25T02:25:06.156909+0000","last_fullsized":"2023-08-25T02:25:06.156909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:58.916927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.217095+0000","last_change":"2023-08-25T02:25:03.613384+0000","last_active":"2023-08-25T02:25:04.217095+0000","last_peered":"2023-08-25T02:25:04.217095+0000","last_clean":"2023-08-25T02:25:04.217095+0000","last_became_active":"2023-08-25T02:25:03.613097+0000","last_became_peered":"2023-08-25T02:25:03.613097+0000","last_unstale":"2023-08-25T02:25:04.217095+0000","last_undegraded":"2023-08-25T02:25:04.217095+0000","last_fullsized":"2023-08-25T02:25:04.217095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:43:03.447001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214227+0000","last_change":"2023-08-25T02:25:04.225009+0000","last_active":"2023-08-25T02:25:05.214227+0000","last_peered":"2023-08-25T02:25:05.214227+0000","last_clean":"2023-08-25T02:25:05.214227+0000","last_became_active":"2023-08-25T02:25:04.224678+0000","last_became_peered":"2023-08-25T02:25:04.224678+0000","last_unstale":"2023-08-25T02:25:05.214227+0000","last_undegraded":"2023-08-25T02:25:05.214227+0000","last_fullsized":"2023-08-25T02:25:05.214227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:45:44.498691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632106+0000","last_change":"2023-08-25T02:25:03.607620+0000","last_active":"2023-08-25T02:25:07.632106+0000","last_peered":"2023-08-25T02:25:07.632106+0000","last_clean":"2023-08-25T02:25:07.632106+0000","last_became_active":"2023-08-25T02:25:03.607329+0000","last_became_peered":"2023-08-25T02:25:03.607329+0000","last_unstale":"2023-08-25T02:25:07.632106+0000","last_undegraded":"2023-08-25T02:25:07.632106+0000","last_fullsized":"2023-08-25T02:25:07.632106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:21.994627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473471+0000","last_change":"2023-08-25T02:24:05.131624+0000","last_active":"2023-08-25T02:25:06.473471+0000","last_peered":"2023-08-25T02:25:06.473471+0000","last_clean":"2023-08-25T02:25:06.473471+0000","last_became_active":"2023-08-25T02:24:05.131084+0000","last_became_peered":"2023-08-25T02:24:05.131084+0000","last_unstale":"2023-08-25T02:25:06.473471+0000","last_undegraded":"2023-08-25T02:25:06.473471+0000","last_fullsized":"2023-08-25T02:25:06.473471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_clean_scrub_stamp":"2023-08-25T02:21:10.049896+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:44.618723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232268+0000","last_change":"2023-08-25T02:25:04.231797+0000","last_active":"2023-08-25T02:25:04.232268+0000","last_peered":"2023-08-25T02:25:04.232268+0000","last_clean":"2023-08-25T02:25:04.232268+0000","last_became_active":"2023-08-25T02:25:04.231520+0000","last_became_peered":"2023-08-25T02:25:04.231520+0000","last_unstale":"2023-08-25T02:25:04.232268+0000","last_undegraded":"2023-08-25T02:25:04.232268+0000","last_fullsized":"2023-08-25T02:25:04.232268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:55:20.388497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473636+0000","last_change":"2023-08-25T02:25:03.611517+0000","last_active":"2023-08-25T02:25:06.473636+0000","last_peered":"2023-08-25T02:25:06.473636+0000","last_clean":"2023-08-25T02:25:06.473636+0000","last_became_active":"2023-08-25T02:25:03.611205+0000","last_became_peered":"2023-08-25T02:25:03.611205+0000","last_unstale":"2023-08-25T02:25:06.473636+0000","last_undegraded":"2023-08-25T02:25:06.473636+0000","last_fullsized":"2023-08-25T02:25:06.473636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:04:13.071834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657665+0000","last_change":"2023-08-25T02:25:03.613795+0000","last_active":"2023-08-25T02:25:06.657665+0000","last_peered":"2023-08-25T02:25:06.657665+0000","last_clean":"2023-08-25T02:25:06.657665+0000","last_became_active":"2023-08-25T02:25:03.613410+0000","last_became_peered":"2023-08-25T02:25:03.613410+0000","last_unstale":"2023-08-25T02:25:06.657665+0000","last_undegraded":"2023-08-25T02:25:06.657665+0000","last_fullsized":"2023-08-25T02:25:06.657665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:47:37.646320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.232803+0000","last_change":"2023-08-25T02:25:04.232186+0000","last_active":"2023-08-25T02:25:04.232803+0000","last_peered":"2023-08-25T02:25:04.232803+0000","last_clean":"2023-08-25T02:25:04.232803+0000","last_became_active":"2023-08-25T02:25:04.231853+0000","last_became_peered":"2023-08-25T02:25:04.231853+0000","last_unstale":"2023-08-25T02:25:04.232803+0000","last_undegraded":"2023-08-25T02:25:04.232803+0000","last_fullsized":"2023-08-25T02:25:04.232803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:14:44.584942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.219541+0000","last_change":"2023-08-25T02:25:03.615541+0000","last_active":"2023-08-25T02:25:04.219541+0000","last_peered":"2023-08-25T02:25:04.219541+0000","last_clean":"2023-08-25T02:25:04.219541+0000","last_became_active":"2023-08-25T02:25:03.615196+0000","last_became_peered":"2023-08-25T02:25:03.615196+0000","last_unstale":"2023-08-25T02:25:04.219541+0000","last_undegraded":"2023-08-25T02:25:04.219541+0000","last_fullsized":"2023-08-25T02:25:04.219541+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:06:02.183736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214288+0000","last_change":"2023-08-25T02:25:04.227688+0000","last_active":"2023-08-25T02:25:05.214288+0000","last_peered":"2023-08-25T02:25:05.214288+0000","last_clean":"2023-08-25T02:25:05.214288+0000","last_became_active":"2023-08-25T02:25:04.227394+0000","last_became_peered":"2023-08-25T02:25:04.227394+0000","last_unstale":"2023-08-25T02:25:05.214288+0000","last_undegraded":"2023-08-25T02:25:05.214288+0000","last_fullsized":"2023-08-25T02:25:05.214288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:53:57.237749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633564+0000","last_change":"2023-08-25T02:25:03.608500+0000","last_active":"2023-08-25T02:25:07.633564+0000","last_peered":"2023-08-25T02:25:07.633564+0000","last_clean":"2023-08-25T02:25:07.633564+0000","last_became_active":"2023-08-25T02:25:03.608172+0000","last_became_peered":"2023-08-25T02:25:03.608172+0000","last_unstale":"2023-08-25T02:25:07.633564+0000","last_undegraded":"2023-08-25T02:25:07.633564+0000","last_fullsized":"2023-08-25T02:25:07.633564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:18:45.063308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938372+0000","last_change":"2023-08-25T02:25:04.226768+0000","last_active":"2023-08-25T02:25:05.938372+0000","last_peered":"2023-08-25T02:25:05.938372+0000","last_clean":"2023-08-25T02:25:05.938372+0000","last_became_active":"2023-08-25T02:25:04.226459+0000","last_became_peered":"2023-08-25T02:25:04.226459+0000","last_unstale":"2023-08-25T02:25:05.938372+0000","last_undegraded":"2023-08-25T02:25:05.938372+0000","last_fullsized":"2023-08-25T02:25:05.938372+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:04:52.374035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438230+0000","last_change":"2023-08-25T02:25:03.612306+0000","last_active":"2023-08-25T02:25:06.438230+0000","last_peered":"2023-08-25T02:25:06.438230+0000","last_clean":"2023-08-25T02:25:06.438230+0000","last_became_active":"2023-08-25T02:25:03.611922+0000","last_became_peered":"2023-08-25T02:25:03.611922+0000","last_unstale":"2023-08-25T02:25:06.438230+0000","last_undegraded":"2023-08-25T02:25:06.438230+0000","last_fullsized":"2023-08-25T02:25:06.438230+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:14:09.728422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657329+0000","last_change":"2023-08-25T02:25:04.235032+0000","last_active":"2023-08-25T02:25:06.657329+0000","last_peered":"2023-08-25T02:25:06.657329+0000","last_clean":"2023-08-25T02:25:06.657329+0000","last_became_active":"2023-08-25T02:25:04.234626+0000","last_became_peered":"2023-08-25T02:25:04.234626+0000","last_unstale":"2023-08-25T02:25:06.657329+0000","last_undegraded":"2023-08-25T02:25:06.657329+0000","last_fullsized":"2023-08-25T02:25:06.657329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:00.849934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213814+0000","last_change":"2023-08-25T02:25:03.608869+0000","last_active":"2023-08-25T02:25:05.213814+0000","last_peered":"2023-08-25T02:25:05.213814+0000","last_clean":"2023-08-25T02:25:05.213814+0000","last_became_active":"2023-08-25T02:25:03.608530+0000","last_became_peered":"2023-08-25T02:25:03.608530+0000","last_unstale":"2023-08-25T02:25:05.213814+0000","last_undegraded":"2023-08-25T02:25:05.213814+0000","last_fullsized":"2023-08-25T02:25:05.213814+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:51:39.415744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236891+0000","last_change":"2023-08-25T02:25:04.231830+0000","last_active":"2023-08-25T02:25:04.236891+0000","last_peered":"2023-08-25T02:25:04.236891+0000","last_clean":"2023-08-25T02:25:04.236891+0000","last_became_active":"2023-08-25T02:25:04.231530+0000","last_became_peered":"2023-08-25T02:25:04.231530+0000","last_unstale":"2023-08-25T02:25:04.236891+0000","last_undegraded":"2023-08-25T02:25:04.236891+0000","last_fullsized":"2023-08-25T02:25:04.236891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:04.520766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.220091+0000","last_change":"2023-08-25T02:25:03.610625+0000","last_active":"2023-08-25T02:25:04.220091+0000","last_peered":"2023-08-25T02:25:04.220091+0000","last_clean":"2023-08-25T02:25:04.220091+0000","last_became_active":"2023-08-25T02:25:03.610305+0000","last_became_peered":"2023-08-25T02:25:03.610305+0000","last_unstale":"2023-08-25T02:25:04.220091+0000","last_undegraded":"2023-08-25T02:25:04.220091+0000","last_fullsized":"2023-08-25T02:25:04.220091+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:41:23.669327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214041+0000","last_change":"2023-08-25T02:25:04.234533+0000","last_active":"2023-08-25T02:25:05.214041+0000","last_peered":"2023-08-25T02:25:05.214041+0000","last_clean":"2023-08-25T02:25:05.214041+0000","last_became_active":"2023-08-25T02:25:04.234167+0000","last_became_peered":"2023-08-25T02:25:04.234167+0000","last_unstale":"2023-08-25T02:25:05.214041+0000","last_undegraded":"2023-08-25T02:25:05.214041+0000","last_fullsized":"2023-08-25T02:25:05.214041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:23:55.422360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633060+0000","last_change":"2023-08-25T02:25:03.610139+0000","last_active":"2023-08-25T02:25:07.633060+0000","last_peered":"2023-08-25T02:25:07.633060+0000","last_clean":"2023-08-25T02:25:07.633060+0000","last_became_active":"2023-08-25T02:25:03.609811+0000","last_became_peered":"2023-08-25T02:25:03.609811+0000","last_unstale":"2023-08-25T02:25:07.633060+0000","last_undegraded":"2023-08-25T02:25:07.633060+0000","last_fullsized":"2023-08-25T02:25:07.633060+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:01.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474201+0000","last_change":"2023-08-25T02:25:04.233179+0000","last_active":"2023-08-25T02:25:06.474201+0000","last_peered":"2023-08-25T02:25:06.474201+0000","last_clean":"2023-08-25T02:25:06.474201+0000","last_became_active":"2023-08-25T02:25:04.232874+0000","last_became_peered":"2023-08-25T02:25:04.232874+0000","last_unstale":"2023-08-25T02:25:06.474201+0000","last_undegraded":"2023-08-25T02:25:06.474201+0000","last_fullsized":"2023-08-25T02:25:06.474201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:54.317603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.220736+0000","last_change":"2023-08-25T02:25:03.619258+0000","last_active":"2023-08-25T02:25:04.220736+0000","last_peered":"2023-08-25T02:25:04.220736+0000","last_clean":"2023-08-25T02:25:04.220736+0000","last_became_active":"2023-08-25T02:25:03.618954+0000","last_became_peered":"2023-08-25T02:25:03.618954+0000","last_unstale":"2023-08-25T02:25:04.220736+0000","last_undegraded":"2023-08-25T02:25:04.220736+0000","last_fullsized":"2023-08-25T02:25:04.220736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:20.995650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474059+0000","last_change":"2023-08-25T02:25:04.232620+0000","last_active":"2023-08-25T02:25:06.474059+0000","last_peered":"2023-08-25T02:25:06.474059+0000","last_clean":"2023-08-25T02:25:06.474059+0000","last_became_active":"2023-08-25T02:25:04.232314+0000","last_became_peered":"2023-08-25T02:25:04.232314+0000","last_unstale":"2023-08-25T02:25:06.474059+0000","last_undegraded":"2023-08-25T02:25:06.474059+0000","last_fullsized":"2023-08-25T02:25:06.474059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:39:46.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157074+0000","last_change":"2023-08-25T02:25:03.617794+0000","last_active":"2023-08-25T02:25:06.157074+0000","last_peered":"2023-08-25T02:25:06.157074+0000","last_clean":"2023-08-25T02:25:06.157074+0000","last_became_active":"2023-08-25T02:25:03.617521+0000","last_became_peered":"2023-08-25T02:25:03.617521+0000","last_unstale":"2023-08-25T02:25:06.157074+0000","last_undegraded":"2023-08-25T02:25:06.157074+0000","last_fullsized":"2023-08-25T02:25:06.157074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:35:22.049419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474136+0000","last_change":"2023-08-25T02:25:04.229796+0000","last_active":"2023-08-25T02:25:06.474136+0000","last_peered":"2023-08-25T02:25:06.474136+0000","last_clean":"2023-08-25T02:25:06.474136+0000","last_became_active":"2023-08-25T02:25:04.227073+0000","last_became_peered":"2023-08-25T02:25:04.227073+0000","last_unstale":"2023-08-25T02:25:06.474136+0000","last_undegraded":"2023-08-25T02:25:06.474136+0000","last_fullsized":"2023-08-25T02:25:06.474136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:06:43.826591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.609293+0000","last_change":"2023-08-25T02:25:03.612702+0000","last_active":"2023-08-25T02:25:04.609293+0000","last_peered":"2023-08-25T02:25:04.609293+0000","last_clean":"2023-08-25T02:25:04.609293+0000","last_became_active":"2023-08-25T02:25:03.612482+0000","last_became_peered":"2023-08-25T02:25:03.612482+0000","last_unstale":"2023-08-25T02:25:04.609293+0000","last_undegraded":"2023-08-25T02:25:04.609293+0000","last_fullsized":"2023-08-25T02:25:04.609293+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:48.686491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474025+0000","last_change":"2023-08-25T02:25:04.229892+0000","last_active":"2023-08-25T02:25:06.474025+0000","last_peered":"2023-08-25T02:25:06.474025+0000","last_clean":"2023-08-25T02:25:06.474025+0000","last_became_active":"2023-08-25T02:25:04.229581+0000","last_became_peered":"2023-08-25T02:25:04.229581+0000","last_unstale":"2023-08-25T02:25:06.474025+0000","last_undegraded":"2023-08-25T02:25:06.474025+0000","last_fullsized":"2023-08-25T02:25:06.474025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:15:54.252164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236914+0000","last_change":"2023-08-25T02:25:04.231992+0000","last_active":"2023-08-25T02:25:04.236914+0000","last_peered":"2023-08-25T02:25:04.236914+0000","last_clean":"2023-08-25T02:25:04.236914+0000","last_became_active":"2023-08-25T02:25:04.231676+0000","last_became_peered":"2023-08-25T02:25:04.231676+0000","last_unstale":"2023-08-25T02:25:04.236914+0000","last_undegraded":"2023-08-25T02:25:04.236914+0000","last_fullsized":"2023-08-25T02:25:04.236914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:48:53.083401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230592+0000","last_change":"2023-08-25T02:25:04.227123+0000","last_active":"2023-08-25T02:25:04.230592+0000","last_peered":"2023-08-25T02:25:04.230592+0000","last_clean":"2023-08-25T02:25:04.230592+0000","last_became_active":"2023-08-25T02:25:04.226828+0000","last_became_peered":"2023-08-25T02:25:04.226828+0000","last_unstale":"2023-08-25T02:25:04.230592+0000","last_undegraded":"2023-08-25T02:25:04.230592+0000","last_fullsized":"2023-08-25T02:25:04.230592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:59:31.717429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.233062+0000","last_change":"2023-08-25T02:25:04.232635+0000","last_active":"2023-08-25T02:25:04.233062+0000","last_peered":"2023-08-25T02:25:04.233062+0000","last_clean":"2023-08-25T02:25:04.233062+0000","last_became_active":"2023-08-25T02:25:04.232353+0000","last_became_peered":"2023-08-25T02:25:04.232353+0000","last_unstale":"2023-08-25T02:25:04.233062+0000","last_undegraded":"2023-08-25T02:25:04.233062+0000","last_fullsized":"2023-08-25T02:25:04.233062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:54:26.193989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938254+0000","last_change":"2023-08-25T02:25:04.226977+0000","last_active":"2023-08-25T02:25:05.938254+0000","last_peered":"2023-08-25T02:25:05.938254+0000","last_clean":"2023-08-25T02:25:05.938254+0000","last_became_active":"2023-08-25T02:25:04.226579+0000","last_became_peered":"2023-08-25T02:25:04.226579+0000","last_unstale":"2023-08-25T02:25:05.938254+0000","last_undegraded":"2023-08-25T02:25:05.938254+0000","last_fullsized":"2023-08-25T02:25:05.938254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:00.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.225841+0000","last_change":"2023-08-25T02:25:04.225050+0000","last_active":"2023-08-25T02:25:04.225841+0000","last_peered":"2023-08-25T02:25:04.225841+0000","last_clean":"2023-08-25T02:25:04.225841+0000","last_became_active":"2023-08-25T02:25:04.224720+0000","last_became_peered":"2023-08-25T02:25:04.224720+0000","last_unstale":"2023-08-25T02:25:04.225841+0000","last_undegraded":"2023-08-25T02:25:04.225841+0000","last_fullsized":"2023-08-25T02:25:04.225841+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:29.863461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.230964+0000","last_change":"2023-08-25T02:25:04.228996+0000","last_active":"2023-08-25T02:25:04.230964+0000","last_peered":"2023-08-25T02:25:04.230964+0000","last_clean":"2023-08-25T02:25:04.230964+0000","last_became_active":"2023-08-25T02:25:04.228526+0000","last_became_peered":"2023-08-25T02:25:04.228526+0000","last_unstale":"2023-08-25T02:25:04.230964+0000","last_undegraded":"2023-08-25T02:25:04.230964+0000","last_fullsized":"2023-08-25T02:25:04.230964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:00:39.738094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.236090+0000","last_change":"2023-08-25T02:25:04.234921+0000","last_active":"2023-08-25T02:25:04.236090+0000","last_peered":"2023-08-25T02:25:04.236090+0000","last_clean":"2023-08-25T02:25:04.236090+0000","last_became_active":"2023-08-25T02:25:04.234588+0000","last_became_peered":"2023-08-25T02:25:04.234588+0000","last_unstale":"2023-08-25T02:25:04.236090+0000","last_undegraded":"2023-08-25T02:25:04.236090+0000","last_fullsized":"2023-08-25T02:25:04.236090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:12:21.426235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214180+0000","last_change":"2023-08-25T02:25:04.227310+0000","last_active":"2023-08-25T02:25:05.214180+0000","last_peered":"2023-08-25T02:25:05.214180+0000","last_clean":"2023-08-25T02:25:05.214180+0000","last_became_active":"2023-08-25T02:25:04.226940+0000","last_became_peered":"2023-08-25T02:25:04.226940+0000","last_unstale":"2023-08-25T02:25:05.214180+0000","last_undegraded":"2023-08-25T02:25:05.214180+0000","last_fullsized":"2023-08-25T02:25:05.214180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:21:22.991356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156816+0000","last_change":"2023-08-25T02:25:04.230835+0000","last_active":"2023-08-25T02:25:06.156816+0000","last_peered":"2023-08-25T02:25:06.156816+0000","last_clean":"2023-08-25T02:25:06.156816+0000","last_became_active":"2023-08-25T02:25:04.230534+0000","last_became_peered":"2023-08-25T02:25:04.230534+0000","last_unstale":"2023-08-25T02:25:06.156816+0000","last_undegraded":"2023-08-25T02:25:06.156816+0000","last_fullsized":"2023-08-25T02:25:06.156816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:09.527955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213887+0000","last_change":"2023-08-25T02:25:04.224250+0000","last_active":"2023-08-25T02:25:05.213887+0000","last_peered":"2023-08-25T02:25:05.213887+0000","last_clean":"2023-08-25T02:25:05.213887+0000","last_became_active":"2023-08-25T02:25:04.223591+0000","last_became_peered":"2023-08-25T02:25:04.223591+0000","last_unstale":"2023-08-25T02:25:05.213887+0000","last_undegraded":"2023-08-25T02:25:05.213887+0000","last_fullsized":"2023-08-25T02:25:05.213887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:26:52.465008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438356+0000","last_change":"2023-08-25T02:25:04.229455+0000","last_active":"2023-08-25T02:25:06.438356+0000","last_peered":"2023-08-25T02:25:06.438356+0000","last_clean":"2023-08-25T02:25:06.438356+0000","last_became_active":"2023-08-25T02:25:04.228114+0000","last_became_peered":"2023-08-25T02:25:04.228114+0000","last_unstale":"2023-08-25T02:25:06.438356+0000","last_undegraded":"2023-08-25T02:25:06.438356+0000","last_fullsized":"2023-08-25T02:25:06.438356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:12:29.745750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.227555+0000","last_change":"2023-08-25T02:25:04.226625+0000","last_active":"2023-08-25T02:25:04.227555+0000","last_peered":"2023-08-25T02:25:04.227555+0000","last_clean":"2023-08-25T02:25:04.227555+0000","last_became_active":"2023-08-25T02:25:04.226253+0000","last_became_peered":"2023-08-25T02:25:04.226253+0000","last_unstale":"2023-08-25T02:25:04.227555+0000","last_undegraded":"2023-08-25T02:25:04.227555+0000","last_fullsized":"2023-08-25T02:25:04.227555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:48:06.916471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-08-25T02:25:04.224861+0000","last_change":"2023-08-25T02:25:04.223776+0000","last_active":"2023-08-25T02:25:04.224861+0000","last_peered":"2023-08-25T02:25:04.224861+0000","last_clean":"2023-08-25T02:25:04.224861+0000","last_became_active":"2023-08-25T02:25:04.223597+0000","last_became_peered":"2023-08-25T02:25:04.223597+0000","last_unstale":"2023-08-25T02:25:04.224861+0000","last_undegraded":"2023-08-25T02:25:04.224861+0000","last_fullsized":"2023-08-25T02:25:04.224861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:38:17.316678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3636,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":8546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":831488,"data_stored":393540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":10,"up_from":68,"seq":292057776143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1003361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.37},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.622},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":9,"up_from":63,"seq":270582939668,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":403199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.629}]},{"osd":4,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.50900000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.55000000000000004}]}]},{"osd":8,"up_from":56,"seq":240518168600,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":411813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.745}]},{"osd":7,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":7,"up_from":49,"seq":210453397533,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1454080,"data_stored":1002137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.497}]},{"osd":1,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42599999999999999,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.432},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60199999999999998,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.628}]},{"osd":4,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.77700000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.65200000000000002,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92100000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70499999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.88900000000000001}]},{"osd":8,"last update":"Fri Aug 25 02:24:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.83699999999999997}]},{"osd":9,"last update":"Fri Aug 25 02:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.90900000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.80500000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]},{"osd":6,"up_from":44,"seq":188978561057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":843776,"data_stored":402617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.373},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.38400000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.442,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76000000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72699999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46600000000000003,"15min":0.46100000000000002},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.433},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44400000000000001,"15min":0.443},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59999999999999998,"15min":0.59299999999999997},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.53400000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86499999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.46800000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.62},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.56200000000000006}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.752}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":412526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47999999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.51600000000000001,"15min":0.48999999999999999},"min":{"1min":0.44600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.495,"15min":0.48299999999999998},"min":{"1min":0.37,"5min":0.307,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Aug 25 02:24:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.372,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83399999999999996,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.39300000000000002},"min":{"1min":0.33600000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.45900000000000002,"15min":0.41999999999999998},"min":{"1min":0.36499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.42099999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57299999999999995,"15min":0.56899999999999995},"min":{"1min":0.46700000000000003,"5min":0.441,"15min":0.441},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.629,"15min":0.61199999999999999},"min":{"1min":0.502,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.48799999999999999,"5min":0.434,"15min":0.434},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.621},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.95799999999999996}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.72299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":1.014}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":847872,"data_stored":402707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.434,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Aug 25 02:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.33900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55800000000000005,"15min":0.55000000000000004},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.53700000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.64700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.32400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71699999999999997,"5min":0.754,"15min":0.754},"last":0.47699999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83899999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73399999999999999}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.49299999999999999}]},{"osd":10,"last update":"Fri Aug 25 02:25:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":860160,"data_stored":404263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.499,"15min":0.48399999999999999},"min":{"1min":0.35599999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.48999999999999999,"15min":0.46200000000000002},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45200000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54600000000000004,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.497},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.53300000000000003},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Aug 25 02:24:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76800000000000002,"5min":0.871,"15min":0.871},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.435,"15min":0.42999999999999999},"min":{"1min":0.33700000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.433,"15min":0.42799999999999999},"min":{"1min":0.32900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.378}]},{"osd":6,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64100000000000001},"min":{"1min":0.47299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51700000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.65600000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57999999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":411291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.312},"min":{"1min":0.25800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.311,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.52500000000000002,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.56799999999999995,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.52600000000000002},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.55600000000000005,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Aug 25 02:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.59299999999999997},"min":{"1min":0.46500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53300000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.48299999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.495}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57999999999999996}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.59099999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":860160,"data_stored":411209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.309,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.38300000000000001,"15min":0.36099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.44700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52600000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.442}]},{"osd":6,"last update":"Fri Aug 25 02:24:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56499999999999995}]},{"osd":8,"last update":"Fri Aug 25 02:24:22 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.51600000000000001}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57999999999999996}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.504},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1002700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.496},"min":{"1min":0.36399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.314}]},{"osd":1,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.27600000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56000000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28299999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72199999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.40999999999999998}]},{"osd":4,"last update":"Fri Aug 25 02:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60699999999999998,"15min":0.60499999999999998},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54300000000000004}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.501},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.67600000000000005,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.59199999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.436}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri Aug 25 02:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T02:25:11.113 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-08-25T02:25:11.113 INFO:tasks.cephadm:Setting up client nodes... 2023-08-25T02:25:11.114 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-08-25T02:25:11.692 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:12.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 -- 172.21.15.89:0/1333534062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead40ffb40 msgr2=0x7fead40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:12.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 --2- 172.21.15.89:0/1333534062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead40ffb40 0x7fead40fffc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7feac4009990 tx=0x7feac4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 -- 172.21.15.89:0/1333534062 shutdown_connections 2023-08-25T02:25:12.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 --2- 172.21.15.89:0/1333534062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 0x7fead4107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 --2- 172.21.15.89:0/1333534062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead4100f70 0x7fead4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 --2- 172.21.15.89:0/1333534062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead40ffb40 0x7fead40fffc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 -- 172.21.15.89:0/1333534062 >> 172.21.15.89:0/1333534062 conn(0x7fead40fa5f0 msgr2=0x7fead40fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:12.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 -- 172.21.15.89:0/1333534062 shutdown_connections 2023-08-25T02:25:12.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.938+0000 7fead9b3c700 1 -- 172.21.15.89:0/1333534062 wait complete. 2023-08-25T02:25:12.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 Processor -- start 2023-08-25T02:25:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 -- start start 2023-08-25T02:25:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 0x7fead41966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 0x7fead4196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:12.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:12.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead419e280 con 0x7fead4100f70 2023-08-25T02:25:12.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead419e3c0 con 0x7fead4100500 2023-08-25T02:25:12.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead9b3c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead419e550 con 0x7fead40ffb40 2023-08-25T02:25:12.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46822/0 (socket says 172.21.15.89:46822) 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.939+0000 7fead3fff700 1 -- 172.21.15.89:0/2189377184 learned_addr learned my addr 172.21.15.89:0/2189377184 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead2ffd700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 0x7fead4196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead37fe700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 0x7fead41966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 -- 172.21.15.89:0/2189377184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 msgr2=0x7fead41966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 0x7fead41966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 -- 172.21.15.89:0/2189377184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 msgr2=0x7fead4196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 0x7fead4196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 -- 172.21.15.89:0/2189377184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac40095f0 con 0x7fead4100f70 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead37fe700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 0x7fead41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead3fff700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feacc007f10 tx=0x7feacc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00ddd0 con 0x7fead4100f70 2023-08-25T02:25:12.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead419e830 con 0x7fead4100f70 2023-08-25T02:25:12.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feacc00f040 con 0x7fead4100f70 2023-08-25T02:25:12.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc015400 con 0x7fead4100f70 2023-08-25T02:25:12.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.940+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead419ed30 con 0x7fead4100f70 2023-08-25T02:25:12.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.941+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead4106ef0 con 0x7fead4100f70 2023-08-25T02:25:12.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.942+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7feacc004020 con 0x7fead4100f70 2023-08-25T02:25:12.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.943+0000 7fead0ff9700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 0x7feabc0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:12.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.943+0000 7fead37fe700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 0x7feabc0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:12.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.943+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7feacc0caba0 con 0x7fead4100f70 2023-08-25T02:25:12.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.943+0000 7fead37fe700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 0x7feabc0a2410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feac8009b50 tx=0x7feac8009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:12.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:12.947+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7feacc04ee00 con 0x7fead4100f70 2023-08-25T02:25:12.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:12 smithi121 ceph-mon[100099]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:12.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:12 smithi121 ceph-mon[100099]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:12.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:12 smithi121 ceph-mon[100099]: pgmap v254: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2023-08-25T02:25:12.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:12 smithi122 ceph-mon[100457]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:12.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:12 smithi122 ceph-mon[100457]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:12.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:12 smithi122 ceph-mon[100457]: pgmap v254: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2023-08-25T02:25:13.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:12 smithi089 ceph-mon[98732]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:13.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:12 smithi089 ceph-mon[98732]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:13.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:12 smithi089 ceph-mon[98732]: pgmap v254: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2023-08-25T02:25:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.213+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fead402ccf0 con 0x7fead4100f70 2023-08-25T02:25:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.217+0000 7fead0ff9700 1 -- 172.21.15.89:0/2189377184 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7feacc0949e0 con 0x7fead4100f70 2023-08-25T02:25:13.218 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-08-25T02:25:13.219 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAJEehkD2zLDBAAwIiGQl1ld7ia2aBL96401A== 2023-08-25T02:25:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 msgr2=0x7feabc0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 0x7feabc0a2410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feac8009b50 tx=0x7feac8009550 comp rx=0 tx=0).stop 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 msgr2=0x7fead419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feacc007f10 tx=0x7feacc00d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 shutdown_connections 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fead4100f70 0x7fead419dcb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fead40ffb40 0x7fead41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feabc09ff50 0x7feabc0a2410 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 --2- 172.21.15.89:0/2189377184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fead4100500 0x7fead4196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.219+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 >> 172.21.15.89:0/2189377184 conn(0x7fead40fa5f0 msgr2=0x7fead4104030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.220+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 shutdown_connections 2023-08-25T02:25:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:13.220+0000 7fead9b3c700 1 -- 172.21.15.89:0/2189377184 wait complete. 2023-08-25T02:25:13.633 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:25:13.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-08-25T02:25:13.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-08-25T02:25:13.675 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-08-25T02:25:13.675 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-08-25T02:25:13.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph mgr dump --format=json 2023-08-25T02:25:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:13 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-08-25T02:25:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:13 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-08-25T02:25:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:13 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-08-25T02:25:13.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:13 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-08-25T02:25:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:13 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-08-25T02:25:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:13 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2189377184' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-08-25T02:25:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:14.766 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:14 smithi089 ceph-mon[98732]: pgmap v255: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2023-08-25T02:25:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:14 smithi121 ceph-mon[100099]: pgmap v255: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2023-08-25T02:25:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:14 smithi122 ceph-mon[100457]: pgmap v255: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2023-08-25T02:25:15.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.476+0000 7f130849e700 1 -- 172.21.15.89:0/1404851455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300106a20 msgr2=0x7f1300106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:15.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.476+0000 7f130849e700 1 --2- 172.21.15.89:0/1404851455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300106a20 0x7f1300106e00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0099c0 tx=0x7f12fc009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- 172.21.15.89:0/1404851455 shutdown_connections 2023-08-25T02:25:15.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- 172.21.15.89:0/1404851455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 0x7f130010d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- 172.21.15.89:0/1404851455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300106a20 0x7f1300106e00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- 172.21.15.89:0/1404851455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300068d40 0x7f13000691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- 172.21.15.89:0/1404851455 >> 172.21.15.89:0/1404851455 conn(0x7f1300076740 msgr2=0x7f1300076b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:15.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- 172.21.15.89:0/1404851455 shutdown_connections 2023-08-25T02:25:15.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- 172.21.15.89:0/1404851455 wait complete. 2023-08-25T02:25:15.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 Processor -- start 2023-08-25T02:25:15.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- start start 2023-08-25T02:25:15.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:15.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 0x7f130019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:15.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 0x7f13001968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:15.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1300196f60 con 0x7f1300068d40 2023-08-25T02:25:15.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001970d0 con 0x7f1300106a20 2023-08-25T02:25:15.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130849e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1300197290 con 0x7f1300069700 2023-08-25T02:25:15.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130623a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:15.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130623a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46832/0 (socket says 172.21.15.89:46832) 2023-08-25T02:25:15.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.477+0000 7f130623a700 1 -- 172.21.15.89:0/1579222980 learned_addr learned my addr 172.21.15.89:0/1579222980 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:15.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f1305a39700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 0x7f130019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:15.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f1306a3b700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 0x7f13001968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:15.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 -- 172.21.15.89:0/1579222980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 msgr2=0x7f130019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:15.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 0x7f130019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 -- 172.21.15.89:0/1579222980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 msgr2=0x7f13001968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:15.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 0x7f13001968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 -- 172.21.15.89:0/1579222980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0095f0 con 0x7f1300068d40 2023-08-25T02:25:15.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f1306a3b700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 0x7f13001968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:15.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130623a700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b580 tx=0x7f12fc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:15.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc01d070 con 0x7f1300068d40 2023-08-25T02:25:15.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12fc022470 con 0x7f1300068d40 2023-08-25T02:25:15.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00f6c0 con 0x7f1300068d40 2023-08-25T02:25:15.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1300197510 con 0x7f1300068d40 2023-08-25T02:25:15.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.478+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13001a3810 con 0x7f1300068d40 2023-08-25T02:25:15.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.479+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300068510 con 0x7f1300068d40 2023-08-25T02:25:15.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.480+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f12fc0225e0 con 0x7f1300068d40 2023-08-25T02:25:15.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.480+0000 7f12f77fe700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 0x7f12ec0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:15.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.481+0000 7f1305a39700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 0x7f12ec0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:15.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.481+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f12fc0cccb0 con 0x7f1300068d40 2023-08-25T02:25:15.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.481+0000 7f1305a39700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 0x7f12ec0a2410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f0006fd0 tx=0x7f12f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:15.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.484+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f12fc096cb0 con 0x7f1300068d40 2023-08-25T02:25:15.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.727+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f130004e620 con 0x7f1300068d40 2023-08-25T02:25:15.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.732+0000 7f12f77fe700 1 -- 172.21.15.89:0/1579222980 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+258012 (secure 0 0 0) 0x7f12fc09a2d0 con 0x7f1300068d40 2023-08-25T02:25:15.736 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 msgr2=0x7f12ec0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 0x7f12ec0a2410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f0006fd0 tx=0x7f12f0008040 comp rx=0 tx=0).stop 2023-08-25T02:25:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 msgr2=0x7f130019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b580 tx=0x7f12fc005ef0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 shutdown_connections 2023-08-25T02:25:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f12ec09ff50 0x7f12ec0a2410 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1300106a20 0x7f13001968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300068d40 0x7f130019c820 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 --2- 172.21.15.89:0/1579222980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1300069700 0x7f130019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 >> 172.21.15.89:0/1579222980 conn(0x7f1300076740 msgr2=0x7f1300102490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 shutdown_connections 2023-08-25T02:25:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:15.735+0000 7f130849e700 1 -- 172.21.15.89:0/1579222980 wait complete. 2023-08-25T02:25:16.116 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1949193934},{"type":"v1","addr":"172.21.15.89:6801","nonce":1949193934}]},"active_addr":"172.21.15.89:6801/1949193934","active_change":"2023-08-25T02:19:03.221200+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.89:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":1951762039}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3424586982}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":989947782}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2658035648}]}]} 2023-08-25T02:25:16.123 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-08-25T02:25:16.123 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-08-25T02:25:16.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd dump --format=json 2023-08-25T02:25:16.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:16 smithi121 ceph-mon[100099]: pgmap v256: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2023-08-25T02:25:16.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:16 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1579222980' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-08-25T02:25:16.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:16 smithi122 ceph-mon[100457]: pgmap v256: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2023-08-25T02:25:16.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:16 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1579222980' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-08-25T02:25:16.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:16 smithi089 ceph-mon[98732]: pgmap v256: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 15 op/s 2023-08-25T02:25:16.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:16 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1579222980' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-08-25T02:25:16.675 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 -- 172.21.15.89:0/878247715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 msgr2=0x7f5f2c10c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 --2- 172.21.15.89:0/878247715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c10c760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400b0a0 tx=0x7f5f2400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 -- 172.21.15.89:0/878247715 shutdown_connections 2023-08-25T02:25:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 --2- 172.21.15.89:0/878247715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c105890 0x7f5f2c105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 --2- 172.21.15.89:0/878247715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c10c760 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 --2- 172.21.15.89:0/878247715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c0ffab0 0x7f5f2c0fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 -- 172.21.15.89:0/878247715 >> 172.21.15.89:0/878247715 conn(0x7f5f2c076740 msgr2=0x7f5f2c076b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 -- 172.21.15.89:0/878247715 shutdown_connections 2023-08-25T02:25:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.894+0000 7f5f31992700 1 -- 172.21.15.89:0/878247715 wait complete. 2023-08-25T02:25:17.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 Processor -- start 2023-08-25T02:25:17.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 -- start start 2023-08-25T02:25:17.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c0ffab0 0x7f5f2c19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:17.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:17.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 0x7f5f2c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:17.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c199c80 con 0x7f5f2c100470 2023-08-25T02:25:17.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c199df0 con 0x7f5f2c0ffab0 2023-08-25T02:25:17.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f31992700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c199fb0 con 0x7f5f2c105890 2023-08-25T02:25:17.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f2a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:17.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f2a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46846/0 (socket says 172.21.15.89:46846) 2023-08-25T02:25:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.895+0000 7f5f2a7fc700 1 -- 172.21.15.89:0/2894739614 learned_addr learned my addr 172.21.15.89:0/2894739614 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2b7fe700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 0x7f5f2c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2affd700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c0ffab0 0x7f5f2c19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 -- 172.21.15.89:0/2894739614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 msgr2=0x7f5f2c199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 0x7f5f2c199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 -- 172.21.15.89:0/2894739614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c0ffab0 msgr2=0x7f5f2c19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c0ffab0 0x7f5f2c19da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:17.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 -- 172.21.15.89:0/2894739614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24009d00 con 0x7f5f2c100470 2023-08-25T02:25:17.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2b7fe700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 0x7f5f2c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:17.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f2a7fc700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00b530 tx=0x7f5f1c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:17.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c011830 con 0x7f5f2c100470 2023-08-25T02:25:17.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f1c011e50 con 0x7f5f2c100470 2023-08-25T02:25:17.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c19a290 con 0x7f5f2c100470 2023-08-25T02:25:17.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00f2e0 con 0x7f5f2c100470 2023-08-25T02:25:17.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.896+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c1a5bf0 con 0x7f5f2c100470 2023-08-25T02:25:17.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.897+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c068510 con 0x7f5f2c100470 2023-08-25T02:25:17.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.899+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5f1c010580 con 0x7f5f2c100470 2023-08-25T02:25:17.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.899+0000 7f5f30990700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 0x7f5f140a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:17.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.899+0000 7f5f2affd700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 0x7f5f140a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:17.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.900+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5f1c0cb190 con 0x7f5f2c100470 2023-08-25T02:25:17.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.900+0000 7f5f2affd700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 0x7f5f140a2410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005950 tx=0x7f5f200058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:17.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:17.903+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5f1c095070 con 0x7f5f2c100470 2023-08-25T02:25:18.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.091+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f2c04e620 con 0x7f5f2c100470 2023-08-25T02:25:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.092+0000 7f5f30990700 1 -- 172.21.15.89:0/2894739614 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f5f1c098690 con 0x7f5f2c100470 2023-08-25T02:25:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:18.094 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:18.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 msgr2=0x7f5f140a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:18.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 0x7f5f140a2410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005950 tx=0x7f5f200058e0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 msgr2=0x7f5f2c19dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:18.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00b530 tx=0x7f5f1c00b8f0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 shutdown_connections 2023-08-25T02:25:18.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f1409ff50 0x7f5f140a2410 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f2c0ffab0 0x7f5f2c19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f2c105890 0x7f5f2c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.094+0000 7f5f31992700 1 --2- 172.21.15.89:0/2894739614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f2c100470 0x7f5f2c19dfd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:18.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.095+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 >> 172.21.15.89:0/2894739614 conn(0x7f5f2c076740 msgr2=0x7f5f2c0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:18.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.095+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 shutdown_connections 2023-08-25T02:25:18.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:18.095+0000 7f5f31992700 1 -- 172.21.15.89:0/2894739614 wait complete. 2023-08-25T02:25:18.414 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:18 smithi089 ceph-mon[98732]: pgmap v257: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2023-08-25T02:25:18.415 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:18 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2894739614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:18.416 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-08-25T02:25:18.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd dump --format=json 2023-08-25T02:25:18.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:18 smithi121 ceph-mon[100099]: pgmap v257: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2023-08-25T02:25:18.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:18 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2894739614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:18.777 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:18 smithi122 ceph-mon[100457]: pgmap v257: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2023-08-25T02:25:18.778 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:18 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2894739614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:18.969 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:20.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 -- 172.21.15.89:0/552133785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c0fee10 msgr2=0x7f841c0ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:20.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 --2- 172.21.15.89:0/552133785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c0fee10 0x7f841c0ff290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8408009990 tx=0x7f8408009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 -- 172.21.15.89:0/552133785 shutdown_connections 2023-08-25T02:25:20.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 --2- 172.21.15.89:0/552133785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c105960 0x7f841c105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 --2- 172.21.15.89:0/552133785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 0x7f841c10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 --2- 172.21.15.89:0/552133785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c0fee10 0x7f841c0ff290 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 -- 172.21.15.89:0/552133785 >> 172.21.15.89:0/552133785 conn(0x7f841c075d90 msgr2=0x7f841c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 -- 172.21.15.89:0/552133785 shutdown_connections 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.141+0000 7f8422e53700 1 -- 172.21.15.89:0/552133785 wait complete. 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 Processor -- start 2023-08-25T02:25:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 -- start start 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 0x7f841c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 0x7f841c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c196f90 con 0x7f841c105960 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c197100 con 0x7f841c0fee10 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8422e53700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c1972c0 con 0x7f841c0ff7d0 2023-08-25T02:25:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:20.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46862/0 (socket says 172.21.15.89:46862) 2023-08-25T02:25:20.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 -- 172.21.15.89:0/2686158722 learned_addr learned my addr 172.21.15.89:0/2686158722 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:20.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f841bfff700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 0x7f841c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:20.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8420bef700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 0x7f841c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:20.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 -- 172.21.15.89:0/2686158722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 msgr2=0x7f841c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:20.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 0x7f841c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 -- 172.21.15.89:0/2686158722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 msgr2=0x7f841c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:20.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 0x7f841c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 -- 172.21.15.89:0/2686158722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080095f0 con 0x7f841c105960 2023-08-25T02:25:20.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f8420bef700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 0x7f841c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:20.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.142+0000 7f84213f0700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f841400b9e0 tx=0x7f841400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:20.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.143+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841400c9b0 con 0x7f841c105960 2023-08-25T02:25:20.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.143+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c1975a0 con 0x7f841c105960 2023-08-25T02:25:20.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.143+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8414014440 con 0x7f841c105960 2023-08-25T02:25:20.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.143+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84140125c0 con 0x7f841c105960 2023-08-25T02:25:20.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.143+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c1a3890 con 0x7f841c105960 2023-08-25T02:25:20.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.144+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c04e620 con 0x7f841c105960 2023-08-25T02:25:20.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.145+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f84140145b0 con 0x7f841c105960 2023-08-25T02:25:20.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.145+0000 7f8419ffb700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 0x7f840c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:20.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.145+0000 7f8420bef700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 0x7f840c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:20.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.145+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8414006e40 con 0x7f841c105960 2023-08-25T02:25:20.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.146+0000 7f8420bef700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 0x7f840c0a2410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8410005fd0 tx=0x7f8410005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:20.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.150+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f84140946d0 con 0x7f841c105960 2023-08-25T02:25:20.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.346+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f841c110a00 con 0x7f841c105960 2023-08-25T02:25:20.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.347+0000 7f8419ffb700 1 -- 172.21.15.89:0/2686158722 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f8414097cf0 con 0x7f841c105960 2023-08-25T02:25:20.349 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:20.349 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:20.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 msgr2=0x7f840c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 0x7f840c0a2410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8410005fd0 tx=0x7f8410005ee0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 msgr2=0x7f841c196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f841400b9e0 tx=0x7f841400bda0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 shutdown_connections 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f840c09ff50 0x7f840c0a2410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c0fee10 0x7f841c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f841c105960 0x7f841c196920 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 --2- 172.21.15.89:0/2686158722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f841c0ff7d0 0x7f841c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:20.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 >> 172.21.15.89:0/2686158722 conn(0x7f841c075d90 msgr2=0x7f841c0fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:20.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 shutdown_connections 2023-08-25T02:25:20.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:20.350+0000 7f8422e53700 1 -- 172.21.15.89:0/2686158722 wait complete. 2023-08-25T02:25:20.692 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:20 smithi089 ceph-mon[98732]: pgmap v258: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-08-25T02:25:20.692 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:20 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2686158722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:20.714 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-08-25T02:25:20.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-08-25T02:25:20.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-08-25T02:25:20.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-08-25T02:25:20.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-08-25T02:25:20.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-08-25T02:25:20.718 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-08-25T02:25:20.718 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-08-25T02:25:20.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-08-25T02:25:20.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-08-25T02:25:20.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-08-25T02:25:20.721 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-08-25T02:25:20.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:20 smithi121 ceph-mon[100099]: pgmap v258: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-08-25T02:25:20.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:20 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2686158722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:20 smithi122 ceph-mon[100457]: pgmap v258: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-08-25T02:25:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:20 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2686158722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:22.109 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.224 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.264 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.376 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.498 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:22 smithi089 ceph-mon[98732]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2023-08-25T02:25:22.697 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.700 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:22.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:22 smithi121 ceph-mon[100099]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2023-08-25T02:25:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:22 smithi122 ceph-mon[100457]: pgmap v259: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2023-08-25T02:25:23.631 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:23.741 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:24.171 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:24.420 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:24.544 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:24 smithi089 ceph-mon[98732]: pgmap v260: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2023-08-25T02:25:24.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:24 smithi121 ceph-mon[100099]: pgmap v260: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2023-08-25T02:25:24.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:24 smithi122 ceph-mon[100457]: pgmap v260: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 8 op/s 2023-08-25T02:25:25.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 -- 172.21.15.89:0/2639320481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 msgr2=0x7ff074068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 --2- 172.21.15.89:0/2639320481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff074068cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff0700099c0 tx=0x7ff070009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 -- 172.21.15.89:0/2639320481 shutdown_connections 2023-08-25T02:25:25.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 --2- 172.21.15.89:0/2639320481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff074105d50 0x7ff07410d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 --2- 172.21.15.89:0/2639320481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff074068cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 --2- 172.21.15.89:0/2639320481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff074105390 0x7ff074105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 -- 172.21.15.89:0/2639320481 >> 172.21.15.89:0/2639320481 conn(0x7ff074076920 msgr2=0x7ff074076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:25.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 -- 172.21.15.89:0/2639320481 shutdown_connections 2023-08-25T02:25:25.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.415+0000 7ff07b61e700 1 -- 172.21.15.89:0/2639320481 wait complete. 2023-08-25T02:25:25.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 Processor -- start 2023-08-25T02:25:25.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 -- start start 2023-08-25T02:25:25.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff07419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:25.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff074105390 0x7ff07419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:25.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff074105d50 0x7ff0741968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:25.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074196f20 con 0x7ff0740688e0 2023-08-25T02:25:25.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074197090 con 0x7ff074105d50 2023-08-25T02:25:25.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff07b61e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074197250 con 0x7ff074105390 2023-08-25T02:25:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff0793ba700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff07419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff0793ba700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff07419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46870/0 (socket says 172.21.15.89:46870) 2023-08-25T02:25:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff0793ba700 1 -- 172.21.15.89:0/3005475926 learned_addr learned my addr 172.21.15.89:0/3005475926 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff079bbb700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff074105d50 0x7ff0741968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.416+0000 7ff078bb9700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff074105390 0x7ff07419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 -- 172.21.15.89:0/3005475926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff074105390 msgr2=0x7ff07419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff074105390 0x7ff07419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 -- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff074105d50 msgr2=0x7ff0741968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff074105d50 0x7ff0741968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700095f0 con 0x7ff0740688e0 2023-08-25T02:25:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff0793ba700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff07419c830 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff07000b580 tx=0x7ff070005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001d070 con 0x7ff0740688e0 2023-08-25T02:25:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070003f90 con 0x7ff0740688e0 2023-08-25T02:25:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070022d20 con 0x7ff0740688e0 2023-08-25T02:25:25.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0741974d0 con 0x7ff0740688e0 2023-08-25T02:25:25.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.417+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0741a3820 con 0x7ff0740688e0 2023-08-25T02:25:25.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.418+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff074197750 con 0x7ff0740688e0 2023-08-25T02:25:25.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.419+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7ff070022470 con 0x7ff0740688e0 2023-08-25T02:25:25.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff06a7fc700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06009ff50 0x7ff0600a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:25.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff078bb9700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06009ff50 0x7ff0600a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:25.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff0700ccdd0 con 0x7ff0740688e0 2023-08-25T02:25:25.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff06a7fc700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] conn(0x7ff0600a9730 0x7ff0600abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0600ac200 con 0x7ff0600a9730 2023-08-25T02:25:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.420+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0700cd220 con 0x7ff0740688e0 2023-08-25T02:25:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.421+0000 7ff078bb9700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06009ff50 0x7ff0600a2410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff074106cb0 tx=0x7ff0640073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.421+0000 7ff079bbb700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] conn(0x7ff0600a9730 0x7ff0600abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.421+0000 7ff079bbb700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] conn(0x7ff0600a9730 0x7ff0600abb50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.422+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== osd.4 v2:172.21.15.121:6808/4228849553 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff0600ac200 con 0x7ff0600a9730 2023-08-25T02:25:25.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.440+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 --> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff0740fe0a0 con 0x7ff0600a9730 2023-08-25T02:25:25.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff06a7fc700 1 -- 172.21.15.89:0/3005475926 <== osd.4 v2:172.21.15.121:6808/4228849553 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff0740fe0a0 con 0x7ff0600a9730 2023-08-25T02:25:25.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] conn(0x7ff0600a9730 msgr2=0x7ff0600abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] conn(0x7ff0600a9730 0x7ff0600abb50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06009ff50 msgr2=0x7ff0600a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06009ff50 0x7ff0600a2410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff074106cb0 tx=0x7ff0640073d0 comp rx=0 tx=0).stop 2023-08-25T02:25:25.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 msgr2=0x7ff07419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:25.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 --2- 172.21.15.89:0/3005475926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0740688e0 0x7ff07419c830 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff07000b580 tx=0x7ff070005740 comp rx=0 tx=0).stop 2023-08-25T02:25:25.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff079bbb700 1 -- 172.21.15.89:0/3005475926 reap_dead start 2023-08-25T02:25:25.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 shutdown_connections 2023-08-25T02:25:25.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.441+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 >> 172.21.15.89:0/3005475926 conn(0x7ff074076920 msgr2=0x7ff0740fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:25.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.442+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 shutdown_connections 2023-08-25T02:25:25.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:25.442+0000 7ff07b61e700 1 -- 172.21.15.89:0/3005475926 wait complete. 2023-08-25T02:25:26.369 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:26 smithi089 ceph-mon[98732]: pgmap v261: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-08-25T02:25:26.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:26 smithi121 ceph-mon[100099]: pgmap v261: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-08-25T02:25:26.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:26 smithi122 ceph-mon[100457]: pgmap v261: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-08-25T02:25:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 -- 172.21.15.89:0/2753762699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec0690a0 msgr2=0x7f09ec069480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 --2- 172.21.15.89:0/2753762699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec0690a0 0x7f09ec069480 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009990 tx=0x7f09e0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 -- 172.21.15.89:0/2753762699 shutdown_connections 2023-08-25T02:25:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 --2- 172.21.15.89:0/2753762699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 0x7f09ec10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 --2- 172.21.15.89:0/2753762699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec0690a0 0x7f09ec069480 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 --2- 172.21.15.89:0/2753762699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec102470 0x7f09ec1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 -- 172.21.15.89:0/2753762699 >> 172.21.15.89:0/2753762699 conn(0x7f09ec076910 msgr2=0x7f09ec076d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 -- 172.21.15.89:0/2753762699 shutdown_connections 2023-08-25T02:25:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.644+0000 7f09f40ce700 1 -- 172.21.15.89:0/2753762699 wait complete. 2023-08-25T02:25:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 Processor -- start 2023-08-25T02:25:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 -- start start 2023-08-25T02:25:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec0690a0 0x7f09ec0ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec102470 0x7f09ec0ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 0x7f09ec100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1a4ee0 con 0x7f09ec102470 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1a5050 con 0x7f09ec0690a0 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f40ce700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1a5210 con 0x7f09ec102e30 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f1e6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec0690a0 0x7f09ec0ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f266b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 0x7f09ec100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f1e6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec0690a0 0x7f09ec0ff8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:37312/0 (socket says 172.21.15.89:37312) 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.645+0000 7f09f1e6a700 1 -- 172.21.15.89:0/4113179941 learned_addr learned my addr 172.21.15.89:0/4113179941 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 -- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec0690a0 msgr2=0x7f09ec0ff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ec0690a0 0x7f09ec0ff8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 -- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec102470 msgr2=0x7f09ec0ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ec102470 0x7f09ec0ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00095f0 con 0x7f09ec102e30 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f266b700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 0x7f09ec100390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09e400efc0 tx=0x7f09e400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4018070 con 0x7f09ec102e30 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec1a5490 con 0x7f09ec102e30 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09e400f040 con 0x7f09ec102e30 2023-08-25T02:25:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e400cdf0 con 0x7f09ec102e30 2023-08-25T02:25:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.646+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec1a5990 con 0x7f09ec102e30 2023-08-25T02:25:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.647+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09d0000ff0 con 0x7f09ec102e30 2023-08-25T02:25:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.648+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f09e4007500 con 0x7f09ec102e30 2023-08-25T02:25:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09deffd700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f09d809ff50 0x7f09d80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09f1e6a700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f09d809ff50 0x7f09d80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f09e40cb9e0 con 0x7f09ec102e30 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09deffd700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] conn(0x7f09d80a9730 0x7f09d80abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09d80ac200 con 0x7f09d80a9730 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09e40cbe30 con 0x7f09ec102e30 2023-08-25T02:25:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.649+0000 7f09f1e6a700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f09d809ff50 0x7f09d80a2410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09e0006010 tx=0x7f09e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.650+0000 7f09f1669700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] conn(0x7f09d80a9730 0x7f09d80abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.650+0000 7f09f1669700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] conn(0x7f09d80a9730 0x7f09d80abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.651+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== osd.1 v2:172.21.15.121:6800/1465063129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f09d80ac200 con 0x7f09d80a9730 2023-08-25T02:25:26.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.667+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 --> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09d0002d70 con 0x7f09d80a9730 2023-08-25T02:25:26.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.667+0000 7f09deffd700 1 -- 172.21.15.89:0/4113179941 <== osd.1 v2:172.21.15.121:6800/1465063129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09d0002d70 con 0x7f09d80a9730 2023-08-25T02:25:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] conn(0x7f09d80a9730 msgr2=0x7f09d80abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] conn(0x7f09d80a9730 0x7f09d80abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f09d809ff50 msgr2=0x7f09d80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f09d809ff50 0x7f09d80a2410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09e0006010 tx=0x7f09e0005960 comp rx=0 tx=0).stop 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 msgr2=0x7f09ec100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 --2- 172.21.15.89:0/4113179941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09ec102e30 0x7f09ec100390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09e400efc0 tx=0x7f09e400c3a0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f266b700 1 -- 172.21.15.89:0/4113179941 reap_dead start 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 shutdown_connections 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 >> 172.21.15.89:0/4113179941 conn(0x7f09ec076910 msgr2=0x7f09ec0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 shutdown_connections 2023-08-25T02:25:26.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.668+0000 7f09f40ce700 1 -- 172.21.15.89:0/4113179941 wait complete. 2023-08-25T02:25:26.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 -- 172.21.15.89:0/3977073139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec00ff860 msgr2=0x7feec00ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 --2- 172.21.15.89:0/3977073139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec00ff860 0x7feec00ffce0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009990 tx=0x7feeb4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 -- 172.21.15.89:0/3977073139 shutdown_connections 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 --2- 172.21.15.89:0/3977073139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec0100220 0x7feec010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 --2- 172.21.15.89:0/3977073139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec0104730 0x7feec0104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 --2- 172.21.15.89:0/3977073139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec00ff860 0x7feec00ffce0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 -- 172.21.15.89:0/3977073139 >> 172.21.15.89:0/3977073139 conn(0x7feec0076920 msgr2=0x7feec0076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 -- 172.21.15.89:0/3977073139 shutdown_connections 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.324+0000 7feec6718700 1 -- 172.21.15.89:0/3977073139 wait complete. 2023-08-25T02:25:26.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 Processor -- start 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 -- start start 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00ff860 0x7feec0198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 0x7feec0199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec0104730 0x7feec019d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019dce0 con 0x7feec0100220 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019de50 con 0x7feec00ff860 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feec6718700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019e010 con 0x7feec0104730 2023-08-25T02:25:26.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feebf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 0x7feec0199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feebf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 0x7feec0199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46888/0 (socket says 172.21.15.89:46888) 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feebf7fe700 1 -- 172.21.15.89:0/3216118787 learned_addr learned my addr 172.21.15.89:0/3216118787 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.325+0000 7feebffff700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00ff860 0x7feec0198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 -- 172.21.15.89:0/3216118787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec0104730 msgr2=0x7feec019d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feec0104730 0x7feec019d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 -- 172.21.15.89:0/3216118787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00ff860 msgr2=0x7feec0198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00ff860 0x7feec0198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb40095f0 con 0x7feec0100220 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebffff700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00ff860 0x7feec0198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebf7fe700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 0x7feec0199260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feeb4000c00 tx=0x7feeb400f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb401d070 con 0x7feec0100220 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeb400fce0 con 0x7feec0100220 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec019e290 con 0x7feec0100220 2023-08-25T02:25:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4017770 con 0x7feec0100220 2023-08-25T02:25:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.326+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec01a5b50 con 0x7feec0100220 2023-08-25T02:25:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.327+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feec004e620 con 0x7feec0100220 2023-08-25T02:25:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.328+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7feeb40178d0 con 0x7feec0100220 2023-08-25T02:25:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.328+0000 7feebd7fa700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feea809ffd0 0x7feea80a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.328+0000 7feebffff700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feea809ffd0 0x7feea80a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.328+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7feeb40cce30 con 0x7feec0100220 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feebd7fa700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] conn(0x7feea80a97b0 0x7feea80abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feea80ac280 con 0x7feea80a97b0 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb40cd1e0 con 0x7feec0100220 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feec4cb5700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] conn(0x7feea80a97b0 0x7feea80abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:26.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feebffff700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feea809ffd0 0x7feea80a2490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feeb0007900 tx=0x7feeb0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.329+0000 7feec4cb5700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] conn(0x7feea80a97b0 0x7feea80abbd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.330+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== osd.0 v2:172.21.15.89:6802/729401604 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7feea80ac280 con 0x7feea80a97b0 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.346+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 --> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feec0110ca0 con 0x7feea80a97b0 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feebd7fa700 1 -- 172.21.15.89:0/3216118787 <== osd.0 v2:172.21.15.89:6802/729401604 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feeb800a720 con 0x7feea80a97b0 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] conn(0x7feea80a97b0 msgr2=0x7feea80abbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] conn(0x7feea80a97b0 0x7feea80abbd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feea809ffd0 msgr2=0x7feea80a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7feea809ffd0 0x7feea80a2490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feeb0007900 tx=0x7feeb0008040 comp rx=0 tx=0).stop 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 msgr2=0x7feec0199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 --2- 172.21.15.89:0/3216118787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feec0100220 0x7feec0199260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feeb4000c00 tx=0x7feeb400f830 comp rx=0 tx=0).stop 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec4cb5700 1 -- 172.21.15.89:0/3216118787 reap_dead start 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 shutdown_connections 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 >> 172.21.15.89:0/3216118787 conn(0x7feec0076920 msgr2=0x7feec00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 shutdown_connections 2023-08-25T02:25:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:26.347+0000 7feec6718700 1 -- 172.21.15.89:0/3216118787 wait complete. 2023-08-25T02:25:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.151+0000 7f0ef907a700 1 -- 172.21.15.89:0/895428369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 msgr2=0x7f0ef40ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.151+0000 7f0ef907a700 1 --2- 172.21.15.89:0/895428369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef40ffce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009990 tx=0x7f0ee4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 -- 172.21.15.89:0/895428369 shutdown_connections 2023-08-25T02:25:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 --2- 172.21.15.89:0/895428369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4100220 0x7f0ef410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 --2- 172.21.15.89:0/895428369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 0x7f0ef4104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 --2- 172.21.15.89:0/895428369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef40ffce0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 -- 172.21.15.89:0/895428369 >> 172.21.15.89:0/895428369 conn(0x7f0ef4076920 msgr2=0x7f0ef4076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 -- 172.21.15.89:0/895428369 shutdown_connections 2023-08-25T02:25:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 -- 172.21.15.89:0/895428369 wait complete. 2023-08-25T02:25:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.152+0000 7f0ef907a700 1 Processor -- start 2023-08-25T02:25:27.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 -- start start 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef4198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4100220 0x7f0ef4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 0x7f0ef419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419dce0 con 0x7f0ef40ff860 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419de50 con 0x7f0ef4100220 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef907a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419e010 con 0x7f0ef4104730 2023-08-25T02:25:27.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef4198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef4198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46918/0 (socket says 172.21.15.89:46918) 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 -- 172.21.15.89:0/3612805363 learned_addr learned my addr 172.21.15.89:0/3612805363 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef259c700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4100220 0x7f0ef4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef359e700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 0x7f0ef419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 -- 172.21.15.89:0/3612805363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 msgr2=0x7f0ef419d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 0x7f0ef419d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 -- 172.21.15.89:0/3612805363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4100220 msgr2=0x7f0ef4199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4100220 0x7f0ef4199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee40095f0 con 0x7f0ef40ff860 2023-08-25T02:25:27.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef359e700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef4104730 0x7f0ef419d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:27.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0ef2d9d700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef4198d20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b530 tx=0x7f0ee800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.153+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8011830 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.154+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ee8011e50 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.154+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef419e2f0 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.154+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800f2e0 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.154+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef41a5b50 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.154+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ef410ea00 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0ee8010380 con 0x7f0ef40ff860 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0edc09ff30 0x7f0edc0a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0ef259c700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0edc09ff30 0x7f0edc0a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0ee80cb290 con 0x7f0ef40ff860 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] conn(0x7f0edc0a9710 0x7f0edc0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0edc0ac1e0 con 0x7f0edc0a9710 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.156+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ee80cb6e0 con 0x7f0ef40ff860 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.157+0000 7f0ef359e700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] conn(0x7f0edc0a9710 0x7f0edc0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.157+0000 7f0ef259c700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0edc09ff30 0x7f0edc0a23f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ef419a340 tx=0x7f0ee4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.157+0000 7f0ef359e700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] conn(0x7f0edc0a9710 0x7f0edc0abb30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.158+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== osd.6 v2:172.21.15.89:6818/1372241573 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f0edc0ac1e0 con 0x7f0edc0a9710 2023-08-25T02:25:27.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 --> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ef404e620 con 0x7f0edc0a9710 2023-08-25T02:25:27.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0edbfff700 1 -- 172.21.15.89:0/3612805363 <== osd.6 v2:172.21.15.89:6818/1372241573 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0ef404e620 con 0x7f0edc0a9710 2023-08-25T02:25:27.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] conn(0x7f0edc0a9710 msgr2=0x7f0edc0abb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] conn(0x7f0edc0a9710 0x7f0edc0abb30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0edc09ff30 msgr2=0x7f0edc0a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0edc09ff30 0x7f0edc0a23f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ef419a340 tx=0x7f0ee4005fd0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 msgr2=0x7f0ef4198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 --2- 172.21.15.89:0/3612805363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef40ff860 0x7f0ef4198d20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b530 tx=0x7f0ee800b840 comp rx=0 tx=0).stop 2023-08-25T02:25:27.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef359e700 1 -- 172.21.15.89:0/3612805363 reap_dead start 2023-08-25T02:25:27.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 shutdown_connections 2023-08-25T02:25:27.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.174+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 >> 172.21.15.89:0/3612805363 conn(0x7f0ef4076920 msgr2=0x7f0ef40fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:27.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.175+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 shutdown_connections 2023-08-25T02:25:27.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.175+0000 7f0ef907a700 1 -- 172.21.15.89:0/3612805363 wait complete. 2023-08-25T02:25:27.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- 172.21.15.89:0/1722964785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb710101af0 msgr2=0x7fb710101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 --2- 172.21.15.89:0/1722964785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb710101af0 0x7fb710101ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb70000b0a0 tx=0x7fb70000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- 172.21.15.89:0/1722964785 shutdown_connections 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 --2- 172.21.15.89:0/1722964785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb71010af20 0x7fb71010d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 --2- 172.21.15.89:0/1722964785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb710101af0 0x7fb710101ed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 --2- 172.21.15.89:0/1722964785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb710102410 0x7fb71010a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- 172.21.15.89:0/1722964785 >> 172.21.15.89:0/1722964785 conn(0x7fb7100fb320 msgr2=0x7fb7100fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- 172.21.15.89:0/1722964785 shutdown_connections 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- 172.21.15.89:0/1722964785 wait complete. 2023-08-25T02:25:27.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 Processor -- start 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.368+0000 7fb717867700 1 -- start start 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb710101af0 0x7fb7100ff750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb710102410 0x7fb7100ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 0x7fb710100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101a5030 con 0x7fb71010af20 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101a51a0 con 0x7fb710101af0 2023-08-25T02:25:27.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb717867700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101a5360 con 0x7fb710102410 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 0x7fb710100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 0x7fb710100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46938/0 (socket says 172.21.15.89:46938) 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 -- 172.21.15.89:0/4224077185 learned_addr learned my addr 172.21.15.89:0/4224077185 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb714e02700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb710102410 0x7fb7100ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715603700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb710101af0 0x7fb7100ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 -- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb710102410 msgr2=0x7fb7100ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb710102410 0x7fb7100ffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 -- 172.21.15.89:0/4224077185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb710101af0 msgr2=0x7fb7100ff750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb710101af0 0x7fb7100ff750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700009d00 con 0x7fb71010af20 2023-08-25T02:25:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb715e04700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 0x7fb710100210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d980 tx=0x7fb70800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708004a30 con 0x7fb71010af20 2023-08-25T02:25:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.369+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7080054e0 con 0x7fb71010af20 2023-08-25T02:25:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.370+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7101a5640 con 0x7fb71010af20 2023-08-25T02:25:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.370+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70800fbc0 con 0x7fb71010af20 2023-08-25T02:25:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.370+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7101a5af0 con 0x7fb71010af20 2023-08-25T02:25:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.370+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb71004e620 con 0x7fb71010af20 2023-08-25T02:25:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fb708004c20 con 0x7fb71010af20 2023-08-25T02:25:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb6f809ff50 0x7fb6f80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb715603700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb6f809ff50 0x7fb6f80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb7080cb360 con 0x7fb71010af20 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] conn(0x7fb6f80a9730 0x7fb6f80abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6f80ac200 con 0x7fb6f80a9730 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7080cb7b0 con 0x7fb71010af20 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.372+0000 7fb715603700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb6f809ff50 0x7fb6f80a2410 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb700009fd0 tx=0x7fb70000b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.373+0000 7fb714e02700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] conn(0x7fb6f80a9730 0x7fb6f80abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.373+0000 7fb714e02700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] conn(0x7fb6f80a9730 0x7fb6f80abb50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.374+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== osd.8 v2:172.21.15.122:6816/1782298471 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fb6f80ac200 con 0x7fb6f80a9730 2023-08-25T02:25:27.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.390+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 --> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb710110bb0 con 0x7fb6f80a9730 2023-08-25T02:25:27.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.390+0000 7fb70e7fc700 1 -- 172.21.15.89:0/4224077185 <== osd.8 v2:172.21.15.122:6816/1782298471 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb704009980 con 0x7fb6f80a9730 2023-08-25T02:25:27.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] conn(0x7fb6f80a9730 msgr2=0x7fb6f80abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] conn(0x7fb6f80a9730 0x7fb6f80abb50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:27.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb6f809ff50 msgr2=0x7fb6f80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb6f809ff50 0x7fb6f80a2410 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb700009fd0 tx=0x7fb70000b700 comp rx=0 tx=0).stop 2023-08-25T02:25:27.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 msgr2=0x7fb710100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:27.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 --2- 172.21.15.89:0/4224077185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb71010af20 0x7fb710100210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d980 tx=0x7fb70800dc90 comp rx=0 tx=0).stop 2023-08-25T02:25:27.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb715e04700 1 -- 172.21.15.89:0/4224077185 reap_dead start 2023-08-25T02:25:27.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 shutdown_connections 2023-08-25T02:25:27.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 >> 172.21.15.89:0/4224077185 conn(0x7fb7100fb320 msgr2=0x7fb71010d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:27.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 shutdown_connections 2023-08-25T02:25:27.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:27.391+0000 7fb717867700 1 -- 172.21.15.89:0/4224077185 wait complete. 2023-08-25T02:25:27.954 INFO:teuthology.orchestra.run.smithi089.stdout:137438953518 2023-08-25T02:25:27.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-08-25T02:25:28.142 INFO:teuthology.orchestra.run.smithi089.stdout:240518168604 2023-08-25T02:25:28.142 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-08-25T02:25:28.449 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:28 smithi089 ceph-mon[98732]: pgmap v262: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-08-25T02:25:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- 172.21.15.89:0/3362360282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338106980 msgr2=0x7ff338106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 --2- 172.21.15.89:0/3362360282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338106980 0x7ff338106d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009990 tx=0x7ff32c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- 172.21.15.89:0/3362360282 shutdown_connections 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 --2- 172.21.15.89:0/3362360282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff338101340 0x7ff33810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 --2- 172.21.15.89:0/3362360282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338106980 0x7ff338106d60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 --2- 172.21.15.89:0/3362360282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff338100980 0x7ff338100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- 172.21.15.89:0/3362360282 >> 172.21.15.89:0/3362360282 conn(0x7ff3380fc450 msgr2=0x7ff3380fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- 172.21.15.89:0/3362360282 shutdown_connections 2023-08-25T02:25:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- 172.21.15.89:0/3362360282 wait complete. 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 Processor -- start 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.312+0000 7ff33eb40700 1 -- start start 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff338100980 0x7ff33819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 0x7ff33819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff338106980 0x7ff338196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338196ee0 con 0x7ff338101340 2023-08-25T02:25:28.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338197050 con 0x7ff338100980 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33eb40700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338197210 con 0x7ff338106980 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 0x7ff33819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 0x7ff33819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46952/0 (socket says 172.21.15.89:46952) 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 -- 172.21.15.89:0/2564034256 learned_addr learned my addr 172.21.15.89:0/2564034256 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33c8dc700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff338100980 0x7ff33819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff33d0dd700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff338106980 0x7ff338196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:28.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 -- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff338106980 msgr2=0x7ff338196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff338106980 0x7ff338196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 -- 172.21.15.89:0/2564034256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff338100980 msgr2=0x7ff33819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff338100980 0x7ff33819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0095f0 con 0x7ff338101340 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.313+0000 7ff337fff700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 0x7ff33819cd30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff32400d950 tx=0x7ff32400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.314+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324009940 con 0x7ff338101340 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.314+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3381974f0 con 0x7ff338101340 2023-08-25T02:25:28.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.314+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff324010460 con 0x7ff338101340 2023-08-25T02:25:28.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.314+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400f620 con 0x7ff338101340 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.314+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3381a37e0 con 0x7ff338101340 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.315+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff33804e620 con 0x7ff338101340 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.316+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7ff3240108d0 con 0x7ff338101340 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff335ffb700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff32809ff50 0x7ff3280a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff33c8dc700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff32809ff50 0x7ff3280a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff3240cb450 con 0x7ff338101340 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff335ffb700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] conn(0x7ff3280a9730 0x7ff3280abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3280ac200 con 0x7ff3280a9730 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3240cb8a0 con 0x7ff338101340 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff33c8dc700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff32809ff50 0x7ff3280a2410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff32c006010 tx=0x7ff32c00b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.317+0000 7ff33d0dd700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] conn(0x7ff3280a9730 0x7ff3280abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.318+0000 7ff33d0dd700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] conn(0x7ff3280a9730 0x7ff3280abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.319+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== osd.5 v2:172.21.15.122:6808/1949734782 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff3280ac200 con 0x7ff3280a9730 2023-08-25T02:25:28.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.337+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 --> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3380fd350 con 0x7ff3280a9730 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff335ffb700 1 -- 172.21.15.89:0/2564034256 <== osd.5 v2:172.21.15.122:6808/1949734782 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff3380fd350 con 0x7ff3280a9730 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] conn(0x7ff3280a9730 msgr2=0x7ff3280abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] conn(0x7ff3280a9730 0x7ff3280abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff32809ff50 msgr2=0x7ff3280a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff32809ff50 0x7ff3280a2410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff32c006010 tx=0x7ff32c00b560 comp rx=0 tx=0).stop 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 msgr2=0x7ff33819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:28.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 --2- 172.21.15.89:0/2564034256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff338101340 0x7ff33819cd30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff32400d950 tx=0x7ff32400dc60 comp rx=0 tx=0).stop 2023-08-25T02:25:28.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33d0dd700 1 -- 172.21.15.89:0/2564034256 reap_dead start 2023-08-25T02:25:28.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 shutdown_connections 2023-08-25T02:25:28.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 >> 172.21.15.89:0/2564034256 conn(0x7ff3380fc450 msgr2=0x7ff3380fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:28.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 shutdown_connections 2023-08-25T02:25:28.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.338+0000 7ff33eb40700 1 -- 172.21.15.89:0/2564034256 wait complete. 2023-08-25T02:25:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:28 smithi121 ceph-mon[100099]: pgmap v262: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-08-25T02:25:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:28 smithi122 ceph-mon[100457]: pgmap v262: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-08-25T02:25:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.906+0000 7f3a0dae8700 1 -- 172.21.15.89:0/3339794563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 msgr2=0x7f3a0810c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/3339794563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0810c6c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000b0a0 tx=0x7f3a0000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 -- 172.21.15.89:0/3339794563 shutdown_connections 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/3339794563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a08104730 0x7f3a08104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/3339794563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0810c6c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/3339794563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a080ff860 0x7f3a080ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 -- 172.21.15.89:0/3339794563 >> 172.21.15.89:0/3339794563 conn(0x7f3a08076920 msgr2=0x7f3a08076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 -- 172.21.15.89:0/3339794563 shutdown_connections 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 -- 172.21.15.89:0/3339794563 wait complete. 2023-08-25T02:25:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 Processor -- start 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.907+0000 7f3a0dae8700 1 -- start start 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080ff860 0x7f3a0819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a08104730 0x7f3a08195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08195e90 con 0x7f3a08100220 2023-08-25T02:25:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08196000 con 0x7f3a080ff860 2023-08-25T02:25:29.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081961c0 con 0x7f3a08104730 2023-08-25T02:25:29.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:29.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0819bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46966/0 (socket says 172.21.15.89:46966) 2023-08-25T02:25:29.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 -- 172.21.15.89:0/266980100 learned_addr learned my addr 172.21.15.89:0/266980100 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:29.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a077fe700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080ff860 0x7f3a0819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a07fff700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a08104730 0x7f3a08195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 -- 172.21.15.89:0/266980100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a08104730 msgr2=0x7f3a08195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a08104730 0x7f3a08195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 -- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080ff860 msgr2=0x7f3a0819b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080ff860 0x7f3a0819b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009d00 con 0x7f3a08100220 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a06ffd700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0819bce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00d950 tx=0x7f39fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:29.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc009940 con 0x7f3a08100220 2023-08-25T02:25:29.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a081964a0 con 0x7f3a08100220 2023-08-25T02:25:29.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39fc010460 con 0x7f3a08100220 2023-08-25T02:25:29.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc00f620 con 0x7f3a08100220 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.908+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a081a2790 con 0x7f3a08100220 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.909+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a0804e620 con 0x7f3a08100220 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f39fc0108d0 con 0x7f3a08100220 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39f009ff50 0x7f39f00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a077fe700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39f009ff50 0x7f39f00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:29.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f39fc0cb3f0 con 0x7f3a08100220 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] conn(0x7f39f00a9730 0x7f39f00abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39f00ac200 con 0x7f39f00a9730 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39fc0cb840 con 0x7f3a08100220 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.911+0000 7f3a077fe700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39f009ff50 0x7f39f00a2410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f39f8005950 tx=0x7f39f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.912+0000 7f3a07fff700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] conn(0x7f39f00a9730 0x7f39f00abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.912+0000 7f3a07fff700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] conn(0x7f39f00a9730 0x7f39f00abb50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.913+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== osd.10 v2:172.21.15.121:6824/3780412782 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f39f00ac200 con 0x7f39f00a9730 2023-08-25T02:25:29.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.929+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 --> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a0810fa90 con 0x7f39f00a9730 2023-08-25T02:25:29.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.929+0000 7f3a04ff9700 1 -- 172.21.15.89:0/266980100 <== osd.10 v2:172.21.15.121:6824/3780412782 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3a000094d0 con 0x7f39f00a9730 2023-08-25T02:25:29.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.929+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] conn(0x7f39f00a9730 msgr2=0x7f39f00abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.929+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] conn(0x7f39f00a9730 0x7f39f00abb50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:29.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39f009ff50 msgr2=0x7f39f00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39f009ff50 0x7f39f00a2410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f39f8005950 tx=0x7f39f800b410 comp rx=0 tx=0).stop 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 msgr2=0x7f3a0819bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 --2- 172.21.15.89:0/266980100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a08100220 0x7f3a0819bce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00d950 tx=0x7f39fc00dc60 comp rx=0 tx=0).stop 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a07fff700 1 -- 172.21.15.89:0/266980100 reap_dead start 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 shutdown_connections 2023-08-25T02:25:29.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 >> 172.21.15.89:0/266980100 conn(0x7f3a08076920 msgr2=0x7f3a080fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:29.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 shutdown_connections 2023-08-25T02:25:29.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:28.930+0000 7f3a0dae8700 1 -- 172.21.15.89:0/266980100 wait complete. 2023-08-25T02:25:29.427 INFO:teuthology.orchestra.run.smithi089.stdout:158913789993 2023-08-25T02:25:29.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-08-25T02:25:29.698 INFO:teuthology.orchestra.run.smithi089.stdout:34359738432 2023-08-25T02:25:29.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-08-25T02:25:30.097 INFO:teuthology.orchestra.run.smithi089.stdout:292057776148 2023-08-25T02:25:30.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-08-25T02:25:30.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 -- 172.21.15.89:0/392636148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 msgr2=0x7f6438100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 --2- 172.21.15.89:0/392636148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f6438100e00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f642c009990 tx=0x7f642c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 -- 172.21.15.89:0/392636148 shutdown_connections 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 --2- 172.21.15.89:0/392636148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6438101340 0x7f643810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 --2- 172.21.15.89:0/392636148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6438106980 0x7f6438106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 --2- 172.21.15.89:0/392636148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f6438100e00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 -- 172.21.15.89:0/392636148 >> 172.21.15.89:0/392636148 conn(0x7f64380fc470 msgr2=0x7f64380fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:30.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 -- 172.21.15.89:0/392636148 shutdown_connections 2023-08-25T02:25:30.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.376+0000 7f6440289700 1 -- 172.21.15.89:0/392636148 wait complete. 2023-08-25T02:25:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.377+0000 7f6440289700 1 Processor -- start 2023-08-25T02:25:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.377+0000 7f6440289700 1 -- start start 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.377+0000 7f6440289700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f643819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6438101340 0x7f643819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6438106980 0x7f64381989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6438199040 con 0x7f6438100980 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64381991b0 con 0x7f6438101340 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6438199370 con 0x7f6438106980 2023-08-25T02:25:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f643819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f643819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46976/0 (socket says 172.21.15.89:46976) 2023-08-25T02:25:30.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 -- 172.21.15.89:0/1296534900 learned_addr learned my addr 172.21.15.89:0/1296534900 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:30.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e826700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6438106980 0x7f64381989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643d824700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6438101340 0x7f643819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 -- 172.21.15.89:0/1296534900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6438106980 msgr2=0x7f64381989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6438106980 0x7f64381989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 -- 172.21.15.89:0/1296534900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6438101340 msgr2=0x7f643819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6438101340 0x7f643819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0095f0 con 0x7f6438100980 2023-08-25T02:25:30.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f643e025700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f643819e950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f64280060b0 tx=0x7f642800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642800dd20 con 0x7f6438100980 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f642800f040 con 0x7f6438100980 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6438199650 con 0x7f6438100980 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428014600 con 0x7f6438100980 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.378+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64381a5940 con 0x7f6438100980 2023-08-25T02:25:30.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.379+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f643804e620 con 0x7f6438100980 2023-08-25T02:25:30.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.380+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6428015400 con 0x7f6438100980 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f64377fe700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f642009ff50 0x7f64200a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f643d824700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f642009ff50 0x7f64200a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f64280cadd0 con 0x7f6438100980 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f64377fe700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] conn(0x7f64200a9730 0x7f64200abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64200ac200 con 0x7f64200a9730 2023-08-25T02:25:30.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64280cb220 con 0x7f6438100980 2023-08-25T02:25:30.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f643e826700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] conn(0x7f64200a9730 0x7f64200abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.381+0000 7f643d824700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f642009ff50 0x7f64200a2410 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6438106050 tx=0x7f642c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.382+0000 7f643e826700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] conn(0x7f64200a9730 0x7f64200abb50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.382+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== osd.3 v2:172.21.15.89:6810/705728932 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f64200ac200 con 0x7f64200a9730 2023-08-25T02:25:30.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.399+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 --> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f643810d740 con 0x7f64200a9730 2023-08-25T02:25:30.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.399+0000 7f64377fe700 1 -- 172.21.15.89:0/1296534900 <== osd.3 v2:172.21.15.89:6810/705728932 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f643810d740 con 0x7f64200a9730 2023-08-25T02:25:30.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] conn(0x7f64200a9730 msgr2=0x7f64200abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] conn(0x7f64200a9730 0x7f64200abb50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f642009ff50 msgr2=0x7f64200a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f642009ff50 0x7f64200a2410 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6438106050 tx=0x7f642c005fd0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 msgr2=0x7f643819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 --2- 172.21.15.89:0/1296534900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6438100980 0x7f643819e950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f64280060b0 tx=0x7f642800d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f643e826700 1 -- 172.21.15.89:0/1296534900 reap_dead start 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 shutdown_connections 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 >> 172.21.15.89:0/1296534900 conn(0x7f64380fc470 msgr2=0x7f643810cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:30.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 shutdown_connections 2023-08-25T02:25:30.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.400+0000 7f6440289700 1 -- 172.21.15.89:0/1296534900 wait complete. 2023-08-25T02:25:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:30 smithi089 ceph-mon[98732]: pgmap v263: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-08-25T02:25:30.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:30 smithi121 ceph-mon[100099]: pgmap v263: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-08-25T02:25:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:30 smithi122 ceph-mon[100457]: pgmap v263: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-08-25T02:25:30.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- 172.21.15.89:0/227501884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a180688e0 msgr2=0x7f8a18068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 --2- 172.21.15.89:0/227501884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a180688e0 0x7f8a18068cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8a080099c0 tx=0x7f8a08009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- 172.21.15.89:0/227501884 shutdown_connections 2023-08-25T02:25:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 --2- 172.21.15.89:0/227501884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a18105d50 0x7f8a1810d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 --2- 172.21.15.89:0/227501884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a180688e0 0x7f8a18068cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 --2- 172.21.15.89:0/227501884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a18105390 0x7f8a18105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- 172.21.15.89:0/227501884 >> 172.21.15.89:0/227501884 conn(0x7f8a18076920 msgr2=0x7f8a18076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- 172.21.15.89:0/227501884 shutdown_connections 2023-08-25T02:25:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- 172.21.15.89:0/227501884 wait complete. 2023-08-25T02:25:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 Processor -- start 2023-08-25T02:25:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.561+0000 7f8a201c8700 1 -- start start 2023-08-25T02:25:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a180688e0 0x7f8a1819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a18105390 0x7f8a1819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 0x7f8a18196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18196e70 con 0x7f8a18105d50 2023-08-25T02:25:30.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18196fe0 con 0x7f8a18105390 2023-08-25T02:25:30.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a181971a0 con 0x7f8a180688e0 2023-08-25T02:25:30.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 0x7f8a18196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 0x7f8a18196800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46996/0 (socket says 172.21.15.89:46996) 2023-08-25T02:25:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 -- 172.21.15.89:0/3825361090 learned_addr learned my addr 172.21.15.89:0/3825361090 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1df64700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a180688e0 0x7f8a1819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1d763700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a18105390 0x7f8a1819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 -- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a180688e0 msgr2=0x7f8a1819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a180688e0 0x7f8a1819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 -- 172.21.15.89:0/3825361090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a18105390 msgr2=0x7f8a1819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a18105390 0x7f8a1819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a080095f0 con 0x7f8a18105d50 2023-08-25T02:25:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a1e765700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 0x7f8a18196800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007f10 tx=0x7f8a1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10017070 con 0x7f8a18105d50 2023-08-25T02:25:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a1000f040 con 0x7f8a18105d50 2023-08-25T02:25:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a18197480 con 0x7f8a18105d50 2023-08-25T02:25:30.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100054b0 con 0x7f8a18105d50 2023-08-25T02:25:30.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.562+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a181a3770 con 0x7f8a18105d50 2023-08-25T02:25:30.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.563+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a1804e620 con 0x7f8a18105d50 2023-08-25T02:25:30.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.564+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f8a10004ad0 con 0x7f8a18105d50 2023-08-25T02:25:30.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a0effd700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8a0409ff50 0x7f8a040a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a1df64700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8a0409ff50 0x7f8a040a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8a100cadc0 con 0x7f8a18105d50 2023-08-25T02:25:30.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a0effd700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] conn(0x7f8a040a9730 0x7f8a040abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:30.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a040ac200 con 0x7f8a040a9730 2023-08-25T02:25:30.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a100cb210 con 0x7f8a18105d50 2023-08-25T02:25:30.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.565+0000 7f8a1df64700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8a0409ff50 0x7f8a040a2410 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a08000c00 tx=0x7f8a080051c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.566+0000 7f8a1d763700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] conn(0x7f8a040a9730 0x7f8a040abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:30.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.566+0000 7f8a1d763700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] conn(0x7f8a040a9730 0x7f8a040abb50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:30.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.567+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== osd.11 v2:172.21.15.122:6824/2395101680 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f8a040ac200 con 0x7f8a040a9730 2023-08-25T02:25:30.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.583+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 --> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a18110b70 con 0x7f8a040a9730 2023-08-25T02:25:30.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a0effd700 1 -- 172.21.15.89:0/3825361090 <== osd.11 v2:172.21.15.122:6824/2395101680 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8a1400a840 con 0x7f8a040a9730 2023-08-25T02:25:30.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] conn(0x7f8a040a9730 msgr2=0x7f8a040abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] conn(0x7f8a040a9730 0x7f8a040abb50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8a0409ff50 msgr2=0x7f8a040a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8a0409ff50 0x7f8a040a2410 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a08000c00 tx=0x7f8a080051c0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 msgr2=0x7f8a18196800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.584+0000 7f8a201c8700 1 --2- 172.21.15.89:0/3825361090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a18105d50 0x7f8a18196800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007f10 tx=0x7f8a1000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:30.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.585+0000 7f8a1e765700 1 -- 172.21.15.89:0/3825361090 reap_dead start 2023-08-25T02:25:30.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.585+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 shutdown_connections 2023-08-25T02:25:30.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.585+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 >> 172.21.15.89:0/3825361090 conn(0x7f8a18076920 msgr2=0x7f8a180fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:30.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.585+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 shutdown_connections 2023-08-25T02:25:30.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:30.585+0000 7f8a201c8700 1 -- 172.21.15.89:0/3825361090 wait complete. 2023-08-25T02:25:30.963 INFO:teuthology.orchestra.run.smithi089.stdout:188978561062 2023-08-25T02:25:30.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-08-25T02:25:30.982 INFO:teuthology.orchestra.run.smithi089.stdout:55834574905 2023-08-25T02:25:30.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-08-25T02:25:31.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/1212047403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 msgr2=0x7fd1e0104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/1212047403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0104b10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009990 tx=0x7fd1d0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/1212047403 shutdown_connections 2023-08-25T02:25:31.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/1212047403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 0x7fd1e010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/1212047403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0104b10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/1212047403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e00ff860 0x7fd1e00ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/1212047403 >> 172.21.15.89:0/1212047403 conn(0x7fd1e0076920 msgr2=0x7fd1e0076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:31.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/1212047403 shutdown_connections 2023-08-25T02:25:31.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.244+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/1212047403 wait complete. 2023-08-25T02:25:31.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 Processor -- start 2023-08-25T02:25:31.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 -- start start 2023-08-25T02:25:31.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e00ff860 0x7fd1e019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 0x7fd1e019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0195eb0 con 0x7fd1e0104730 2023-08-25T02:25:31.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0196020 con 0x7fd1e0100220 2023-08-25T02:25:31.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.245+0000 7fd1e7e9e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01961e0 con 0x7fd1e00ff860 2023-08-25T02:25:31.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47012/0 (socket says 172.21.15.89:47012) 2023-08-25T02:25:31.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 -- 172.21.15.89:0/317418997 learned_addr learned my addr 172.21.15.89:0/317418997 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:31.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e5c3a700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e00ff860 0x7fd1e019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e5439700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 0x7fd1e019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 -- 172.21.15.89:0/317418997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e00ff860 msgr2=0x7fd1e019b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e00ff860 0x7fd1e019b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 -- 172.21.15.89:0/317418997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 msgr2=0x7fd1e019bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 0x7fd1e019bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00095f0 con 0x7fd1e0104730 2023-08-25T02:25:31.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e5439700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1e0100220 0x7fd1e019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:31.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e643b700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0195840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b9e0 tx=0x7fd1d800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800c950 con 0x7fd1e0104730 2023-08-25T02:25:31.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e01964c0 con 0x7fd1e0104730 2023-08-25T02:25:31.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1d8014440 con 0x7fd1e0104730 2023-08-25T02:25:31.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80075d0 con 0x7fd1e0104730 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.246+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e01a27b0 con 0x7fd1e0104730 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.247+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1e004e620 con 0x7fd1e0104730 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.248+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd1d80145b0 con 0x7fd1e0104730 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.248+0000 7fd1d6ffd700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd1cc09ff50 0x7fd1cc0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1e5c3a700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd1cc09ff50 0x7fd1cc0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd1d8006e40 con 0x7fd1e0104730 2023-08-25T02:25:31.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1d6ffd700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] conn(0x7fd1cc0a9730 0x7fd1cc0abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1cc0ac200 con 0x7fd1cc0a9730 2023-08-25T02:25:31.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1d80cb240 con 0x7fd1e0104730 2023-08-25T02:25:31.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1e5439700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] conn(0x7fd1cc0a9730 0x7fd1cc0abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1e5c3a700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd1cc09ff50 0x7fd1cc0a2410 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0006010 tx=0x7fd1d000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.249+0000 7fd1e5439700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] conn(0x7fd1cc0a9730 0x7fd1cc0abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.250+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== osd.9 v2:172.21.15.89:6826/3342522203 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fd1cc0ac200 con 0x7fd1cc0a9730 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.265+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 --> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd1e010fb90 con 0x7fd1cc0a9730 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1d6ffd700 1 -- 172.21.15.89:0/317418997 <== osd.9 v2:172.21.15.89:6826/3342522203 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd1dc009610 con 0x7fd1cc0a9730 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] conn(0x7fd1cc0a9730 msgr2=0x7fd1cc0abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] conn(0x7fd1cc0a9730 0x7fd1cc0abb50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd1cc09ff50 msgr2=0x7fd1cc0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd1cc09ff50 0x7fd1cc0a2410 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0006010 tx=0x7fd1d000b540 comp rx=0 tx=0).stop 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 msgr2=0x7fd1e0195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 --2- 172.21.15.89:0/317418997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1e0104730 0x7fd1e0195840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b9e0 tx=0x7fd1d800bda0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e643b700 1 -- 172.21.15.89:0/317418997 reap_dead start 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 shutdown_connections 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 >> 172.21.15.89:0/317418997 conn(0x7fd1e0076920 msgr2=0x7fd1e00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 shutdown_connections 2023-08-25T02:25:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.266+0000 7fd1e7e9e700 1 -- 172.21.15.89:0/317418997 wait complete. 2023-08-25T02:25:31.589 INFO:teuthology.orchestra.run.smithi089.stdout:107374182450 2023-08-25T02:25:31.590 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-08-25T02:25:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 -- 172.21.15.89:0/3147477156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4104730 msgr2=0x7f81c4104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 --2- 172.21.15.89:0/3147477156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4104730 0x7f81c4104b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009990 tx=0x7f81b4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 -- 172.21.15.89:0/3147477156 shutdown_connections 2023-08-25T02:25:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 --2- 172.21.15.89:0/3147477156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c4100220 0x7f81c410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 --2- 172.21.15.89:0/3147477156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4104730 0x7f81c4104b10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 --2- 172.21.15.89:0/3147477156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81c40ff860 0x7f81c40ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 -- 172.21.15.89:0/3147477156 >> 172.21.15.89:0/3147477156 conn(0x7f81c4076920 msgr2=0x7f81c4076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 -- 172.21.15.89:0/3147477156 shutdown_connections 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.649+0000 7f81c8c23700 1 -- 172.21.15.89:0/3147477156 wait complete. 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 Processor -- start 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 -- start start 2023-08-25T02:25:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81c40ff860 0x7f81c4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 0x7f81c4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c4104730 0x7f81c41a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41988d0 con 0x7f81c4100220 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41a3010 con 0x7f81c4104730 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c8c23700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41a31d0 con 0x7f81c40ff860 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 0x7f81c4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 0x7f81c4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47022/0 (socket says 172.21.15.89:47022) 2023-08-25T02:25:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.650+0000 7f81c1d9b700 1 -- 172.21.15.89:0/58529510 learned_addr learned my addr 172.21.15.89:0/58529510 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c2d9d700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c4104730 0x7f81c41a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c259c700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81c40ff860 0x7f81c4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 -- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81c40ff860 msgr2=0x7f81c4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81c40ff860 0x7f81c4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 -- 172.21.15.89:0/58529510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c4104730 msgr2=0x7f81c41a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c4104730 0x7f81c41a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40095f0 con 0x7f81c4100220 2023-08-25T02:25:31.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c1d9b700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 0x7f81c4197f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81b800d950 tx=0x7f81b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8009940 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81b8010460 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b800f620 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c41a34b0 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.651+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c41a3960 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.655+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f81c410fa10 con 0x7f81c4100220 2023-08-25T02:25:31.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.655+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f81b80108d0 con 0x7f81c4100220 2023-08-25T02:25:31.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81b37fe700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f81ac09ffd0 0x7f81ac0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81c259c700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f81ac09ffd0 0x7f81ac0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f81b80caf20 con 0x7f81c4100220 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81b37fe700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] conn(0x7f81ac0a97b0 0x7f81ac0abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81ac0ac280 con 0x7f81ac0a97b0 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81b80cd540 con 0x7f81c4100220 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.656+0000 7f81c259c700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f81ac09ffd0 0x7f81ac0a2490 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006010 tx=0x7f81b400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.657+0000 7f81c2d9d700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] conn(0x7f81ac0a97b0 0x7f81ac0abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.657+0000 7f81c2d9d700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] conn(0x7f81ac0a97b0 0x7f81ac0abbd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:31.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.658+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== osd.2 v2:172.21.15.122:6800/1354963983 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f81ac0ac280 con 0x7f81ac0a97b0 2023-08-25T02:25:31.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.687+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 --> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f81c404e620 con 0x7f81ac0a97b0 2023-08-25T02:25:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.687+0000 7f81b37fe700 1 -- 172.21.15.89:0/58529510 <== osd.2 v2:172.21.15.122:6800/1354963983 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f81c404e620 con 0x7f81ac0a97b0 2023-08-25T02:25:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] conn(0x7f81ac0a97b0 msgr2=0x7f81ac0abbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] conn(0x7f81ac0a97b0 0x7f81ac0abbd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f81ac09ffd0 msgr2=0x7f81ac0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f81ac09ffd0 0x7f81ac0a2490 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f81b4006010 tx=0x7f81b400b540 comp rx=0 tx=0).stop 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 msgr2=0x7f81c4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 --2- 172.21.15.89:0/58529510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100220 0x7f81c4197f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81b800d950 tx=0x7f81b800dc60 comp rx=0 tx=0).stop 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c2d9d700 1 -- 172.21.15.89:0/58529510 reap_dead start 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 shutdown_connections 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 >> 172.21.15.89:0/58529510 conn(0x7f81c4076920 msgr2=0x7f81c40fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 shutdown_connections 2023-08-25T02:25:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:31.688+0000 7f81c8c23700 1 -- 172.21.15.89:0/58529510 wait complete. 2023-08-25T02:25:32.291 INFO:teuthology.orchestra.run.smithi089.stdout:270582939674 2023-08-25T02:25:32.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-08-25T02:25:32.350 INFO:teuthology.orchestra.run.smithi089.stdout:77309411384 2023-08-25T02:25:32.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-08-25T02:25:32.520 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:32.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- 172.21.15.89:0/1572176720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0100bb0 msgr2=0x7fc7e0101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 --2- 172.21.15.89:0/1572176720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0100bb0 0x7fc7e0101030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009990 tx=0x7fc7d0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- 172.21.15.89:0/1572176720 shutdown_connections 2023-08-25T02:25:32.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 --2- 172.21.15.89:0/1572176720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 0x7fc7e010d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 --2- 172.21.15.89:0/1572176720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e0106a80 0x7fc7e0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 --2- 172.21.15.89:0/1572176720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0100bb0 0x7fc7e0101030 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- 172.21.15.89:0/1572176720 >> 172.21.15.89:0/1572176720 conn(0x7fc7e0078940 msgr2=0x7fc7e0078d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:32.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- 172.21.15.89:0/1572176720 shutdown_connections 2023-08-25T02:25:32.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- 172.21.15.89:0/1572176720 wait complete. 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 Processor -- start 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.574+0000 7fc7e8274700 1 -- start start 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e0100bb0 0x7fc7e0072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 0x7fc7e0073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 0x7fc7e00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01a4dc0 con 0x7fc7e0106a80 2023-08-25T02:25:32.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01a4f30 con 0x7fc7e0101570 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01a50f0 con 0x7fc7e0100bb0 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 0x7fc7e00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 0x7fc7e00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47038/0 (socket says 172.21.15.89:47038) 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 -- 172.21.15.89:0/660405386 learned_addr learned my addr 172.21.15.89:0/660405386 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6010700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e0100bb0 0x7fc7e0072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e580f700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 0x7fc7e0073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:32.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 -- 172.21.15.89:0/660405386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e0100bb0 msgr2=0x7fc7e0072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e0100bb0 0x7fc7e0072c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 -- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 msgr2=0x7fc7e0073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 0x7fc7e0073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00095f0 con 0x7fc7e0106a80 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e580f700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7e0101570 0x7fc7e0073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e6811700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 0x7fc7e00736e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8007ee0 tx=0x7fc7d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8017070 con 0x7fc7e0106a80 2023-08-25T02:25:32.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e01a53d0 con 0x7fc7e0106a80 2023-08-25T02:25:32.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7d800f040 con 0x7fc7e0106a80 2023-08-25T02:25:32.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.575+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8005740 con 0x7fc7e0106a80 2023-08-25T02:25:32.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.576+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e01a58d0 con 0x7fc7e0106a80 2023-08-25T02:25:32.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.576+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc7e004e620 con 0x7fc7e0106a80 2023-08-25T02:25:32.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.577+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fc7d80058a0 con 0x7fc7e0106a80 2023-08-25T02:25:32.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7d77fe700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc7cc09ff50 0x7fc7cc0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:32.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7e6010700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc7cc09ff50 0x7fc7cc0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:32.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc7d80cb000 con 0x7fc7e0106a80 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7d77fe700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] conn(0x7fc7cc0a9730 0x7fc7cc0abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7cc0ac200 con 0x7fc7cc0a9730 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7d80cb450 con 0x7fc7e0106a80 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7e6010700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc7cc09ff50 0x7fc7cc0a2410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005fd0 tx=0x7fc7dc00af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.578+0000 7fc7e580f700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] conn(0x7fc7cc0a9730 0x7fc7cc0abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.579+0000 7fc7e580f700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] conn(0x7fc7cc0a9730 0x7fc7cc0abb50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.580+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== osd.7 v2:172.21.15.121:6816/62965695 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fc7cc0ac200 con 0x7fc7cc0a9730 2023-08-25T02:25:32.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.596+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 --> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc7e0110be0 con 0x7fc7cc0a9730 2023-08-25T02:25:32.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.596+0000 7fc7d77fe700 1 -- 172.21.15.89:0/660405386 <== osd.7 v2:172.21.15.121:6816/62965695 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc7d001e040 con 0x7fc7cc0a9730 2023-08-25T02:25:32.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] conn(0x7fc7cc0a9730 msgr2=0x7fc7cc0abb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] conn(0x7fc7cc0a9730 0x7fc7cc0abb50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc7cc09ff50 msgr2=0x7fc7cc0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc7cc09ff50 0x7fc7cc0a2410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005fd0 tx=0x7fc7dc00af20 comp rx=0 tx=0).stop 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 msgr2=0x7fc7e00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 --2- 172.21.15.89:0/660405386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7e0106a80 0x7fc7e00736e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8007ee0 tx=0x7fc7d800d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e6811700 1 -- 172.21.15.89:0/660405386 reap_dead start 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 shutdown_connections 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 >> 172.21.15.89:0/660405386 conn(0x7fc7e0078940 msgr2=0x7fc7e0107850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 shutdown_connections 2023-08-25T02:25:32.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:32.597+0000 7fc7e8274700 1 -- 172.21.15.89:0/660405386 wait complete. 2023-08-25T02:25:32.622 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:32 smithi089 ceph-mon[98732]: pgmap v264: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:32.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:32 smithi121 ceph-mon[100099]: pgmap v264: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:32.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:32 smithi122 ceph-mon[100457]: pgmap v264: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:322122547217 2023-08-25T02:25:32.946 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-08-25T02:25:33.170 INFO:teuthology.orchestra.run.smithi089.stdout:210453397539 2023-08-25T02:25:33.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-08-25T02:25:33.303 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.304 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.410 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.562 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.623 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.624 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.857 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:34.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:34 smithi089 ceph-mon[98732]: pgmap v265: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:34.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:34 smithi121 ceph-mon[100099]: pgmap v265: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:34.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:34 smithi122 ceph-mon[100457]: pgmap v265: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:36.297 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:36 smithi089 ceph-mon[98732]: pgmap v266: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:36.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:36 smithi121 ceph-mon[100099]: pgmap v266: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:36.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:36 smithi122 ceph-mon[100457]: pgmap v266: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 -- 172.21.15.89:0/1549179573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc106950 msgr2=0x7f80dc106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 --2- 172.21.15.89:0/1549179573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc106950 0x7f80dc106d30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009990 tx=0x7f80c4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 -- 172.21.15.89:0/1549179573 shutdown_connections 2023-08-25T02:25:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 --2- 172.21.15.89:0/1549179573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc101310 0x7f80dc10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 --2- 172.21.15.89:0/1549179573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc106950 0x7f80dc106d30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 --2- 172.21.15.89:0/1549179573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc100950 0x7f80dc100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 -- 172.21.15.89:0/1549179573 >> 172.21.15.89:0/1549179573 conn(0x7f80dc0fc440 msgr2=0x7f80dc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 -- 172.21.15.89:0/1549179573 shutdown_connections 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 -- 172.21.15.89:0/1549179573 wait complete. 2023-08-25T02:25:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.909+0000 7f80e145e700 1 Processor -- start 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 -- start start 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc100950 0x7f80dc072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc106950 0x7f80dc0736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc1a4e60 con 0x7f80dc101310 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc1a4fd0 con 0x7f80dc106950 2023-08-25T02:25:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80e145e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc1a5190 con 0x7f80dc100950 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47050/0 (socket says 172.21.15.89:47050) 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 -- 172.21.15.89:0/1275994198 learned_addr learned my addr 172.21.15.89:0/1275994198 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80daffd700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc100950 0x7f80dc072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80db7fe700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc106950 0x7f80dc0736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 -- 172.21.15.89:0/1275994198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc100950 msgr2=0x7f80dc072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc100950 0x7f80dc072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 -- 172.21.15.89:0/1275994198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc106950 msgr2=0x7f80dc0736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc106950 0x7f80dc0736a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 -- 172.21.15.89:0/1275994198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40095f0 con 0x7f80dc101310 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80da7fc700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00d950 tx=0x7f80cc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.910+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc009b40 con 0x7f80dc101310 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.911+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc1a5470 con 0x7f80dc101310 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.911+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc1a59c0 con 0x7f80dc101310 2023-08-25T02:25:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.911+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80cc009ca0 con 0x7f80dc101310 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.911+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00f620 con 0x7f80dc101310 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.912+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc005320 con 0x7f80dc101310 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.913+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f80cc010460 con 0x7f80dc101310 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.913+0000 7f80d3fff700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 0x7f80c80a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.913+0000 7f80daffd700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 0x7f80c80a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.913+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f80cc0caf60 con 0x7f80dc101310 2023-08-25T02:25:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.914+0000 7f80daffd700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 0x7f80c80a2490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80c4003730 tx=0x7f80c40036a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:37.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:36.918+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f80cc094e40 con 0x7f80dc101310 2023-08-25T02:25:37.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.119+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f80bc0058a0 con 0x7f80dc101310 2023-08-25T02:25:37.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.119+0000 7f80d3fff700 1 -- 172.21.15.89:0/1275994198 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f80cc098460 con 0x7f80dc101310 2023-08-25T02:25:37.121 INFO:teuthology.orchestra.run.smithi089.stdout:137438953520 2023-08-25T02:25:37.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 msgr2=0x7f80c80a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 0x7f80c80a2490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80c4003730 tx=0x7f80c40036a0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 msgr2=0x7f80dc073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00d950 tx=0x7f80cc00dd10 comp rx=0 tx=0).stop 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 shutdown_connections 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f80c809ffd0 0x7f80c80a2490 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80dc106950 0x7f80dc0736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80dc100950 0x7f80dc072c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 --2- 172.21.15.89:0/1275994198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80dc101310 0x7f80dc073160 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 >> 172.21.15.89:0/1275994198 conn(0x7f80dc0fc440 msgr2=0x7f80dc0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:37.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 shutdown_connections 2023-08-25T02:25:37.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.122+0000 7f80e145e700 1 -- 172.21.15.89:0/1275994198 wait complete. 2023-08-25T02:25:37.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:37 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1275994198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-08-25T02:25:37.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:37 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1275994198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-08-25T02:25:37.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:37 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1275994198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-08-25T02:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 -- 172.21.15.89:0/2082122504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce681058d0 msgr2=0x7fce68105cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 --2- 172.21.15.89:0/2082122504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce681058d0 0x7fce68105cb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fce58009990 tx=0x7fce58009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 -- 172.21.15.89:0/2082122504 shutdown_connections 2023-08-25T02:25:37.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 --2- 172.21.15.89:0/2082122504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce68100450 0x7fce6810c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 --2- 172.21.15.89:0/2082122504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce681058d0 0x7fce68105cb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 --2- 172.21.15.89:0/2082122504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce680ffa90 0x7fce680fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 -- 172.21.15.89:0/2082122504 >> 172.21.15.89:0/2082122504 conn(0x7fce68076750 msgr2=0x7fce68076b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 -- 172.21.15.89:0/2082122504 shutdown_connections 2023-08-25T02:25:37.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.745+0000 7fce70243700 1 -- 172.21.15.89:0/2082122504 wait complete. 2023-08-25T02:25:37.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.746+0000 7fce70243700 1 Processor -- start 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.746+0000 7fce70243700 1 -- start start 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 0x7fce68195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce681058d0 0x7fce6819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6819a930 con 0x7fce680ffa90 2023-08-25T02:25:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6819aaa0 con 0x7fce68100450 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce70243700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6819ac60 con 0x7fce681058d0 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47060/0 (socket says 172.21.15.89:47060) 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 -- 172.21.15.89:0/371065452 learned_addr learned my addr 172.21.15.89:0/371065452 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6d7de700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 0x7fce68195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6e7e0700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce681058d0 0x7fce6819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 -- 172.21.15.89:0/371065452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce681058d0 msgr2=0x7fce6819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce681058d0 0x7fce6819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 -- 172.21.15.89:0/371065452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 msgr2=0x7fce68195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 0x7fce68195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 -- 172.21.15.89:0/371065452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580095f0 con 0x7fce680ffa90 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6d7de700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 0x7fce68195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce6dfdf700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce580095c0 tx=0x7fce5800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.747+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5801d070 con 0x7fce680ffa90 2023-08-25T02:25:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.748+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6819aee0 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.748+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce5800fc00 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.748+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580173f0 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.748+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6819b1b0 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.749+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6804e620 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.750+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fce58022470 con 0x7fce680ffa90 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.750+0000 7fce5effd700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 0x7fce540a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.750+0000 7fce6d7de700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 0x7fce540a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.750+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fce580cc320 con 0x7fce680ffa90 2023-08-25T02:25:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.751+0000 7fce6d7de700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 0x7fce540a2410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce64005950 tx=0x7fce640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.754+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fce58096290 con 0x7fce680ffa90 2023-08-25T02:25:37.952 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:37.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.952+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fce68196ce0 con 0x7fce680ffa90 2023-08-25T02:25:37.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.956+0000 7fce5effd700 1 -- 172.21.15.89:0/371065452 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fce580998b0 con 0x7fce680ffa90 2023-08-25T02:25:37.958 INFO:teuthology.orchestra.run.smithi089.stdout:292057776149 2023-08-25T02:25:37.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 msgr2=0x7fce540a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 0x7fce540a2410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce64005950 tx=0x7fce640058e0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 msgr2=0x7fce68195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce580095c0 tx=0x7fce5800f710 comp rx=0 tx=0).stop 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 shutdown_connections 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce681058d0 0x7fce6819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce680ffa90 0x7fce68195980 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce580095c0 tx=0x7fce5800f710 comp rx=0 tx=0).stop 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fce5409ff50 0x7fce540a2410 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 --2- 172.21.15.89:0/371065452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce68100450 0x7fce68195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:37.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.960+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 >> 172.21.15.89:0/371065452 conn(0x7fce68076750 msgr2=0x7fce680fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:37.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.962+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 shutdown_connections 2023-08-25T02:25:37.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:37.962+0000 7fce70243700 1 -- 172.21.15.89:0/371065452 wait complete. 2023-08-25T02:25:37.984 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953518 got 137438953520 for osd.4 2023-08-25T02:25:37.984 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 -- 172.21.15.89:0/3650733409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940fee70 msgr2=0x7fd3940ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3650733409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940fee70 0x7fd3940ff2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd388009990 tx=0x7fd388009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 -- 172.21.15.89:0/3650733409 shutdown_connections 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3650733409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd394105960 0x7fd394105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3650733409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39410d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3650733409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940fee70 0x7fd3940ff2f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 -- 172.21.15.89:0/3650733409 >> 172.21.15.89:0/3650733409 conn(0x7fd394075d90 msgr2=0x7fd3940fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 -- 172.21.15.89:0/3650733409 shutdown_connections 2023-08-25T02:25:38.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.259+0000 7fd398fc4700 1 -- 172.21.15.89:0/3650733409 wait complete. 2023-08-25T02:25:38.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 Processor -- start 2023-08-25T02:25:38.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 -- start start 2023-08-25T02:25:38.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3940fee70 0x7fd39419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394199100 con 0x7fd394105960 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394199270 con 0x7fd3940fee70 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd398fc4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394199430 con 0x7fd3940ff830 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47076/0 (socket says 172.21.15.89:47076) 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 -- 172.21.15.89:0/3100877217 learned_addr learned my addr 172.21.15.89:0/3100877217 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd39259c700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3940fee70 0x7fd39419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd391d9b700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 -- 172.21.15.89:0/3100877217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 msgr2=0x7fd39419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 -- 172.21.15.89:0/3100877217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3940fee70 msgr2=0x7fd39419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3940fee70 0x7fd39419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd392d9d700 1 -- 172.21.15.89:0/3100877217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3880095f0 con 0x7fd394105960 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.260+0000 7fd391d9b700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd392d9d700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00ada0 tx=0x7fd38c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c00f070 con 0x7fd394105960 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c007b20 con 0x7fd394105960 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c004290 con 0x7fd394105960 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd394199710 con 0x7fd394105960 2023-08-25T02:25:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.261+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3941a5a00 con 0x7fd394105960 2023-08-25T02:25:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.262+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39404e620 con 0x7fd394105960 2023-08-25T02:25:38.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.263+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd38c003a20 con 0x7fd394105960 2023-08-25T02:25:38.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.264+0000 7fd3837fe700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 0x7fd37c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.264+0000 7fd39259c700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 0x7fd37c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.264+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd38c0ca280 con 0x7fd394105960 2023-08-25T02:25:38.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.264+0000 7fd39259c700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 0x7fd37c0a2410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd384007be0 tx=0x7fd3840073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.268+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd38c094160 con 0x7fd394105960 2023-08-25T02:25:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 -- 172.21.15.89:0/1756041975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b4105390 msgr2=0x7f20b4105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 --2- 172.21.15.89:0/1756041975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b4105390 0x7f20b4105810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f20a00099c0 tx=0x7f20a0009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 -- 172.21.15.89:0/1756041975 shutdown_connections 2023-08-25T02:25:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 --2- 172.21.15.89:0/1756041975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b410d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 --2- 172.21.15.89:0/1756041975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b40688e0 0x7f20b4068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 --2- 172.21.15.89:0/1756041975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b4105390 0x7f20b4105810 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 -- 172.21.15.89:0/1756041975 >> 172.21.15.89:0/1756041975 conn(0x7f20b4076920 msgr2=0x7f20b4076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 -- 172.21.15.89:0/1756041975 shutdown_connections 2023-08-25T02:25:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 -- 172.21.15.89:0/1756041975 wait complete. 2023-08-25T02:25:38.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.318+0000 7f20bad25700 1 Processor -- start 2023-08-25T02:25:38.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 -- start start 2023-08-25T02:25:38.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 0x7f20b419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b4105390 0x7f20b419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4197010 con 0x7f20b40688e0 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4197180 con 0x7f20b4105d50 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20bad25700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4197340 con 0x7f20b4105390 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b3fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b4105390 0x7f20b419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:37500/0 (socket says 172.21.15.89:37500) 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 -- 172.21.15.89:0/3713864607 learned_addr learned my addr 172.21.15.89:0/3713864607 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b8ac1700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 0x7f20b419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 -- 172.21.15.89:0/3713864607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b4105390 msgr2=0x7f20b419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b4105390 0x7f20b419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 -- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 msgr2=0x7f20b419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 0x7f20b419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.319+0000 7f20b92c2700 1 -- 172.21.15.89:0/3713864607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00095f0 con 0x7f20b4105d50 2023-08-25T02:25:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.320+0000 7f20b8ac1700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 0x7f20b419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.320+0000 7f20b92c2700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20ac007f10 tx=0x7f20ac00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.322+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac017070 con 0x7f20b4105d50 2023-08-25T02:25:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.322+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20ac00f040 con 0x7f20b4105d50 2023-08-25T02:25:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.322+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac005740 con 0x7f20b4105d50 2023-08-25T02:25:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.322+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b41975c0 con 0x7f20b4105d50 2023-08-25T02:25:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.323+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b40ff9d0 con 0x7f20b4105d50 2023-08-25T02:25:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.324+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b404e620 con 0x7f20b4105d50 2023-08-25T02:25:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.325+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f20ac00dc60 con 0x7f20b4105d50 2023-08-25T02:25:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.326+0000 7f20b1ffb700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 0x7f20a40aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.326+0000 7f20b8ac1700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 0x7f20a40aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.326+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f20ac0ca750 con 0x7f20b4105d50 2023-08-25T02:25:38.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.326+0000 7f20b8ac1700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 0x7f20a40aaca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009be0 tx=0x7f20a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.328 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:38 smithi089 ceph-mon[98732]: pgmap v267: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:38.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:38 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/371065452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-08-25T02:25:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.331+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f20ac094630 con 0x7f20b4105d50 2023-08-25T02:25:38.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:38 smithi121 ceph-mon[100099]: pgmap v267: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:38.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:38 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/371065452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-08-25T02:25:38.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:38 smithi122 ceph-mon[100457]: pgmap v267: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:38.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:38 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/371065452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-08-25T02:25:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.483+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd3941015e0 con 0x7fd394105960 2023-08-25T02:25:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.484+0000 7fd3837fe700 1 -- 172.21.15.89:0/3100877217 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd38c097780 con 0x7fd394105960 2023-08-25T02:25:38.487 INFO:teuthology.orchestra.run.smithi089.stdout:158913789995 2023-08-25T02:25:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 msgr2=0x7fd37c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 0x7fd37c0a2410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd384007be0 tx=0x7fd3840073d0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 msgr2=0x7fd394198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00ada0 tx=0x7fd38c00c330 comp rx=0 tx=0).stop 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 shutdown_connections 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd37c09ff50 0x7fd37c0a2410 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3940fee70 0x7fd39419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd394105960 0x7fd394198a90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 --2- 172.21.15.89:0/3100877217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3940ff830 0x7fd39419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.488+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 >> 172.21.15.89:0/3100877217 conn(0x7fd394075d90 msgr2=0x7fd3941034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.489+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 shutdown_connections 2023-08-25T02:25:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.489+0000 7fd398fc4700 1 -- 172.21.15.89:0/3100877217 wait complete. 2023-08-25T02:25:38.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.580+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f20b4110890 con 0x7f20b4105d50 2023-08-25T02:25:38.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.581+0000 7f20b1ffb700 1 -- 172.21.15.89:0/3713864607 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f20ac0058a0 con 0x7f20b4105d50 2023-08-25T02:25:38.582 INFO:teuthology.orchestra.run.smithi089.stdout:270582939675 2023-08-25T02:25:38.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 msgr2=0x7f20a40aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 0x7f20a40aaca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009be0 tx=0x7f20a8009260 comp rx=0 tx=0).stop 2023-08-25T02:25:38.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 msgr2=0x7f20b4196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20ac007f10 tx=0x7f20ac00d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 shutdown_connections 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f20a40a87e0 0x7f20a40aaca0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20b4105d50 0x7f20b4196860 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b40688e0 0x7f20b419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 --2- 172.21.15.89:0/3713864607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20b4105390 0x7f20b419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 >> 172.21.15.89:0/3713864607 conn(0x7f20b4076920 msgr2=0x7f20b40fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 shutdown_connections 2023-08-25T02:25:38.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.583+0000 7f20bad25700 1 -- 172.21.15.89:0/3713864607 wait complete. 2023-08-25T02:25:38.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.606+0000 7f39c2bd8700 1 -- 172.21.15.89:0/553724450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc101300 msgr2=0x7f39bc10d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.606+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/553724450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc101300 0x7f39bc10d810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009f90 tx=0x7f39b400b330 comp rx=0 tx=0).stop 2023-08-25T02:25:38.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 -- 172.21.15.89:0/553724450 shutdown_connections 2023-08-25T02:25:38.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/553724450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc101300 0x7f39bc10d810 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/553724450 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc106940 0x7f39bc106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/553724450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc100940 0x7f39bc100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 -- 172.21.15.89:0/553724450 >> 172.21.15.89:0/553724450 conn(0x7f39bc0fc410 msgr2=0x7f39bc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 -- 172.21.15.89:0/553724450 shutdown_connections 2023-08-25T02:25:38.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.607+0000 7f39c2bd8700 1 -- 172.21.15.89:0/553724450 wait complete. 2023-08-25T02:25:38.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 Processor -- start 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 -- start start 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 0x7f39bc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc106940 0x7f39bc196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc196fd0 con 0x7f39bc100940 2023-08-25T02:25:38.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc197140 con 0x7f39bc106940 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c2bd8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc197300 con 0x7f39bc101300 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47100/0 (socket says 172.21.15.89:47100) 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 -- 172.21.15.89:0/2014998781 learned_addr learned my addr 172.21.15.89:0/2014998781 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c1175700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc106940 0x7f39bc196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39bbfff700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 0x7f39bc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 -- 172.21.15.89:0/2014998781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 msgr2=0x7f39bc19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 0x7f39bc19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 -- 172.21.15.89:0/2014998781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc106940 msgr2=0x7f39bc196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc106940 0x7f39bc196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 -- 172.21.15.89:0/2014998781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80095f0 con 0x7f39bc100940 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39bbfff700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 0x7f39bc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:38.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.608+0000 7f39c0974700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f39b000d950 tx=0x7f39b000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.609+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0009b40 con 0x7f39bc100940 2023-08-25T02:25:38.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.609+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39b0009ca0 con 0x7f39bc100940 2023-08-25T02:25:38.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.609+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b4009c40 con 0x7f39bc100940 2023-08-25T02:25:38.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.609+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000f620 con 0x7f39bc100940 2023-08-25T02:25:38.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.609+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc1a3700 con 0x7f39bc100940 2023-08-25T02:25:38.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.610+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc04e620 con 0x7f39bc100940 2023-08-25T02:25:38.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.611+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f39b0010460 con 0x7f39bc100940 2023-08-25T02:25:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.611+0000 7f39b9ffb700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 0x7f39ac0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.611+0000 7f39bbfff700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 0x7f39ac0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.611+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f39b00cb4f0 con 0x7f39bc100940 2023-08-25T02:25:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.612+0000 7f39bbfff700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 0x7f39ac0a2490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39bc105fd0 tx=0x7f39a8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.616+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f39b0095330 con 0x7f39bc100940 2023-08-25T02:25:38.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.826+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f39bc0008d0 con 0x7f39bc100940 2023-08-25T02:25:38.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.827+0000 7f39b9ffb700 1 -- 172.21.15.89:0/2014998781 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f39b0098950 con 0x7f39bc100940 2023-08-25T02:25:38.828 INFO:teuthology.orchestra.run.smithi089.stdout:188978561064 2023-08-25T02:25:38.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 msgr2=0x7f39ac0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 0x7f39ac0a2490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39bc105fd0 tx=0x7f39a8009310 comp rx=0 tx=0).stop 2023-08-25T02:25:38.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 msgr2=0x7f39bc19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f39b000d950 tx=0x7f39b000dd10 comp rx=0 tx=0).stop 2023-08-25T02:25:38.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 shutdown_connections 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f39ac09ffd0 0x7f39ac0a2490 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39bc106940 0x7f39bc196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39bc100940 0x7f39bc19c8e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 --2- 172.21.15.89:0/2014998781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39bc101300 0x7f39bc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 >> 172.21.15.89:0/2014998781 conn(0x7f39bc0fc410 msgr2=0x7f39bc0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 shutdown_connections 2023-08-25T02:25:38.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.829+0000 7f39c2bd8700 1 -- 172.21.15.89:0/2014998781 wait complete. 2023-08-25T02:25:38.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 -- 172.21.15.89:0/719378999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c4105860 msgr2=0x7f53c4105c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/719378999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c4105860 0x7f53c4105c40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009990 tx=0x7f53b4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 -- 172.21.15.89:0/719378999 shutdown_connections 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/719378999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/719378999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c4105860 0x7f53c4105c40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/719378999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c40fffe0 0x7f53c4100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 -- 172.21.15.89:0/719378999 >> 172.21.15.89:0/719378999 conn(0x7f53c4075d90 msgr2=0x7f53c40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 -- 172.21.15.89:0/719378999 shutdown_connections 2023-08-25T02:25:38.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.857+0000 7f53ca8f2700 1 -- 172.21.15.89:0/719378999 wait complete. 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 Processor -- start 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 -- start start 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c4105860 0x7f53c4196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c4196e00 con 0x7f53c40fffe0 2023-08-25T02:25:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c4196f70 con 0x7f53c41009a0 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53ca8f2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c4197130 con 0x7f53c4105860 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47114/0 (socket says 172.21.15.89:47114) 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 -- 172.21.15.89:0/3883777525 learned_addr learned my addr 172.21.15.89:0/3883777525 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c37fe700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 -- 172.21.15.89:0/3883777525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c4105860 msgr2=0x7f53c4196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c4105860 0x7f53c4196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 -- 172.21.15.89:0/3883777525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 msgr2=0x7f53c419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c3fff700 1 -- 172.21.15.89:0/3883777525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40095f0 con 0x7f53c40fffe0 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.858+0000 7f53c37fe700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53c3fff700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b401d070 con 0x7f53c40fffe0 2023-08-25T02:25:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53b400be40 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c41973b0 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4022ea0 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.859+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c41a3700 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.860+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c404e620 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.861+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f53b4022470 con 0x7f53c40fffe0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.861+0000 7f53c17fa700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 0x7f53ac0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.861+0000 7f53c37fe700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 0x7f53ac0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:38.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.861+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f53b40cc470 con 0x7f53c40fffe0 2023-08-25T02:25:38.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.862+0000 7f53c37fe700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 0x7f53ac0a2410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53b8005fd0 tx=0x7f53b8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:38.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:38.865+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f53b405ce00 con 0x7f53c40fffe0 2023-08-25T02:25:39.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.072+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f53c4197ca0 con 0x7f53c40fffe0 2023-08-25T02:25:39.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.073+0000 7f53c17fa700 1 -- 172.21.15.89:0/3883777525 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f53b40963e0 con 0x7f53c40fffe0 2023-08-25T02:25:39.074 INFO:teuthology.orchestra.run.smithi089.stdout:240518168606 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 msgr2=0x7f53ac0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 0x7f53ac0a2410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53b8005fd0 tx=0x7f53b8005e30 comp rx=0 tx=0).stop 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 msgr2=0x7f53c419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b400f740 comp rx=0 tx=0).stop 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 shutdown_connections 2023-08-25T02:25:39.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c4105860 0x7f53c4196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c40fffe0 0x7f53c419c710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f53ac09ff50 0x7f53ac0a2410 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 --2- 172.21.15.89:0/3883777525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53c41009a0 0x7f53c419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 >> 172.21.15.89:0/3883777525 conn(0x7f53c4075d90 msgr2=0x7f53c40fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 shutdown_connections 2023-08-25T02:25:39.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.075+0000 7f53ca8f2700 1 -- 172.21.15.89:0/3883777525 wait complete. 2023-08-25T02:25:39.093 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:39.395 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:39 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3100877217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-08-25T02:25:39.396 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:39 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3713864607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-08-25T02:25:39.396 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:39 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2014998781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-08-25T02:25:39.396 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:39 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3883777525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-08-25T02:25:39.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:39 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3100877217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-08-25T02:25:39.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:39 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3713864607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-08-25T02:25:39.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:39 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2014998781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-08-25T02:25:39.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:39 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3883777525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-08-25T02:25:39.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:39 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3100877217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-08-25T02:25:39.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:39 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3713864607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-08-25T02:25:39.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:39 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2014998781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-08-25T02:25:39.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:39 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3883777525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-08-25T02:25:39.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 -- 172.21.15.89:0/3214967191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 msgr2=0x7fec78100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 --2- 172.21.15.89:0/3214967191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78100720 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec74009990 tx=0x7fec74009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 -- 172.21.15.89:0/3214967191 shutdown_connections 2023-08-25T02:25:39.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 --2- 172.21.15.89:0/3214967191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec7810d3b0 0x7fec7810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 --2- 172.21.15.89:0/3214967191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78100720 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 --2- 172.21.15.89:0/3214967191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec78100c60 0x7fec7810ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 -- 172.21.15.89:0/3214967191 >> 172.21.15.89:0/3214967191 conn(0x7fec78076920 msgr2=0x7fec78076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.576+0000 7fec7f812700 1 -- 172.21.15.89:0/3214967191 shutdown_connections 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 -- 172.21.15.89:0/3214967191 wait complete. 2023-08-25T02:25:39.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 Processor -- start 2023-08-25T02:25:39.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 -- start start 2023-08-25T02:25:39.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec78100c60 0x7fec78102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7810d3b0 0x7fec78102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a9290 con 0x7fec78100340 2023-08-25T02:25:39.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a9400 con 0x7fec78100c60 2023-08-25T02:25:39.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7f812700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a95c0 con 0x7fec7810d3b0 2023-08-25T02:25:39.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.577+0000 7fec7d5ae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47128/0 (socket says 172.21.15.89:47128) 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 -- 172.21.15.89:0/269012564 learned_addr learned my addr 172.21.15.89:0/269012564 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7cdad700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec78100c60 0x7fec78102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7ddaf700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7810d3b0 0x7fec78102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 -- 172.21.15.89:0/269012564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7810d3b0 msgr2=0x7fec78102560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7810d3b0 0x7fec78102560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 -- 172.21.15.89:0/269012564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec78100c60 msgr2=0x7fec78102020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec78100c60 0x7fec78102020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 -- 172.21.15.89:0/269012564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740095f0 con 0x7fec78100340 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7d5ae700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec74000c00 tx=0x7fec7400f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7401d070 con 0x7fec78100340 2023-08-25T02:25:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec781a9840 con 0x7fec78100340 2023-08-25T02:25:39.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7400fb80 con 0x7fec78100340 2023-08-25T02:25:39.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74017650 con 0x7fec78100340 2023-08-25T02:25:39.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.578+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec781a9d30 con 0x7fec78100340 2023-08-25T02:25:39.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.579+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7804e620 con 0x7fec78100340 2023-08-25T02:25:39.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.581+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fec74022470 con 0x7fec78100340 2023-08-25T02:25:39.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.582+0000 7fec6e7fc700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 0x7fec640a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.582+0000 7fec7cdad700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 0x7fec640a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.582+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fec740ccd50 con 0x7fec78100340 2023-08-25T02:25:39.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.582+0000 7fec7cdad700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 0x7fec640a2490 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec781032d0 tx=0x7fec6800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:39.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.585+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fec74096c20 con 0x7fec78100340 2023-08-25T02:25:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 -- 172.21.15.89:0/4136832008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105390 msgr2=0x7f696c105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 --2- 172.21.15.89:0/4136832008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105390 0x7f696c105810 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f69600099c0 tx=0x7f6960009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 -- 172.21.15.89:0/4136832008 shutdown_connections 2023-08-25T02:25:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 --2- 172.21.15.89:0/4136832008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c0688e0 0x7f696c068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 --2- 172.21.15.89:0/4136832008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c105d50 0x7f696c10d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 --2- 172.21.15.89:0/4136832008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105390 0x7f696c105810 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 -- 172.21.15.89:0/4136832008 >> 172.21.15.89:0/4136832008 conn(0x7f696c076920 msgr2=0x7f696c076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 -- 172.21.15.89:0/4136832008 shutdown_connections 2023-08-25T02:25:39.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 -- 172.21.15.89:0/4136832008 wait complete. 2023-08-25T02:25:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.628+0000 7f6971267700 1 Processor -- start 2023-08-25T02:25:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 -- start start 2023-08-25T02:25:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c0688e0 0x7f696c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c105390 0x7f696c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1970c0 con 0x7f696c105d50 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c197230 con 0x7f696c105390 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f6971267700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1973f0 con 0x7f696c0688e0 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47146/0 (socket says 172.21.15.89:47146) 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 -- 172.21.15.89:0/2586219738 learned_addr learned my addr 172.21.15.89:0/2586219738 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696affd700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c0688e0 0x7f696c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696a7fc700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c105390 0x7f696c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 -- 172.21.15.89:0/2586219738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c0688e0 msgr2=0x7f696c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c0688e0 0x7f696c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 -- 172.21.15.89:0/2586219738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c105390 msgr2=0x7f696c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c105390 0x7f696c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 -- 172.21.15.89:0/2586219738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69600095f0 con 0x7f696c105d50 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.629+0000 7f696b7fe700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6964007f10 tx=0x7f696400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.630+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964009ea0 con 0x7f696c105d50 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.630+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f696400f040 con 0x7f696c105d50 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.630+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1976d0 con 0x7f696c105d50 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.630+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964004760 con 0x7f696c105d50 2023-08-25T02:25:39.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.630+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1a38a0 con 0x7f696c105d50 2023-08-25T02:25:39.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.631+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c04e620 con 0x7f696c105d50 2023-08-25T02:25:39.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.632+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6964003a00 con 0x7f696c105d50 2023-08-25T02:25:39.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.632+0000 7f6953fff700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 0x7f69540a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:39.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.633+0000 7f696affd700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 0x7f69540a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:39.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.633+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f69640cafe0 con 0x7f696c105d50 2023-08-25T02:25:39.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.633+0000 7f696affd700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 0x7f69540a2410 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f695c005950 tx=0x7f695c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:39.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.636+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6964094ec0 con 0x7f696c105d50 2023-08-25T02:25:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.786+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fec78112b60 con 0x7fec78100340 2023-08-25T02:25:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.787+0000 7fec6e7fc700 1 -- 172.21.15.89:0/269012564 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec7409a240 con 0x7fec78100340 2023-08-25T02:25:39.788 INFO:teuthology.orchestra.run.smithi089.stdout:55834574908 2023-08-25T02:25:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 msgr2=0x7fec640a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 0x7fec640a2490 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec781032d0 tx=0x7fec6800a380 comp rx=0 tx=0).stop 2023-08-25T02:25:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 msgr2=0x7fec78101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec74000c00 tx=0x7fec7400f780 comp rx=0 tx=0).stop 2023-08-25T02:25:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 shutdown_connections 2023-08-25T02:25:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec7810d3b0 0x7fec78102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec78100340 0x7fec78101ae0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fec6409ffd0 0x7fec640a2490 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 --2- 172.21.15.89:0/269012564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec78100c60 0x7fec78102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 >> 172.21.15.89:0/269012564 conn(0x7fec78076920 msgr2=0x7fec7810e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 shutdown_connections 2023-08-25T02:25:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.789+0000 7fec7f812700 1 -- 172.21.15.89:0/269012564 wait complete. 2023-08-25T02:25:39.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.846+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f696c110a10 con 0x7f696c105d50 2023-08-25T02:25:39.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.846+0000 7f6953fff700 1 -- 172.21.15.89:0/2586219738 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f69640984e0 con 0x7f696c105d50 2023-08-25T02:25:39.848 INFO:teuthology.orchestra.run.smithi089.stdout:34359738434 2023-08-25T02:25:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.848+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 msgr2=0x7f69540a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.848+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 0x7f69540a2410 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f695c005950 tx=0x7f695c00b410 comp rx=0 tx=0).stop 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.848+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 msgr2=0x7f696c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.848+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6964007f10 tx=0x7f696400d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 shutdown_connections 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696c105d50 0x7f696c196910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f696c0688e0 0x7f696c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f695409ff50 0x7f69540a2410 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 --2- 172.21.15.89:0/2586219738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f696c105390 0x7f696c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 >> 172.21.15.89:0/2586219738 conn(0x7f696c076920 msgr2=0x7f696c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 shutdown_connections 2023-08-25T02:25:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:39.849+0000 7f6971267700 1 -- 172.21.15.89:0/2586219738 wait complete. 2023-08-25T02:25:39.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789993 got 158913789995 for osd.5 2023-08-25T02:25:39.944 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:40.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:40 smithi089 ceph-mon[98732]: pgmap v268: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-08-25T02:25:40.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:40 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/269012564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-08-25T02:25:40.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:40 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2586219738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-08-25T02:25:40.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:40 smithi121 ceph-mon[100099]: pgmap v268: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-08-25T02:25:40.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:40 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/269012564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-08-25T02:25:40.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:40 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2586219738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-08-25T02:25:40.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:40 smithi122 ceph-mon[100457]: pgmap v268: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-08-25T02:25:40.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:40 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/269012564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-08-25T02:25:40.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:40 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2586219738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-08-25T02:25:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 -- 172.21.15.89:0/3814928527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 msgr2=0x7fde20100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3814928527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde20100db0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fde08009990 tx=0x7fde08009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 -- 172.21.15.89:0/3814928527 shutdown_connections 2023-08-25T02:25:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3814928527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde201012f0 0x7fde2010d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3814928527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde20106930 0x7fde20106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3814928527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde20100db0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 -- 172.21.15.89:0/3814928527 >> 172.21.15.89:0/3814928527 conn(0x7fde200fc420 msgr2=0x7fde200fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 -- 172.21.15.89:0/3814928527 shutdown_connections 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.547+0000 7fde24fc0700 1 -- 172.21.15.89:0/3814928527 wait complete. 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 Processor -- start 2023-08-25T02:25:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 -- start start 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde201012f0 0x7fde2019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 0x7fde20196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.548+0000 7fde24fc0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde20196ea0 con 0x7fde20100930 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde24fc0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde20197010 con 0x7fde20106930 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde24fc0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde201971d0 con 0x7fde201012f0 2023-08-25T02:25:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47152/0 (socket says 172.21.15.89:47152) 2023-08-25T02:25:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 -- 172.21.15.89:0/3136998349 learned_addr learned my addr 172.21.15.89:0/3136998349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1dd9b700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde201012f0 0x7fde2019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1ed9d700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 0x7fde20196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 -- 172.21.15.89:0/3136998349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde201012f0 msgr2=0x7fde2019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde201012f0 0x7fde2019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 -- 172.21.15.89:0/3136998349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 msgr2=0x7fde20196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 0x7fde20196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 -- 172.21.15.89:0/3136998349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080095f0 con 0x7fde20100930 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1ed9d700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 0x7fde20196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde1e59c700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fde180060b0 tx=0x7fde1800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1800ddd0 con 0x7fde20100930 2023-08-25T02:25:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde201974b0 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde1800f040 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18014690 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.549+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde201a37a0 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.550+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2004e620 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.551+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fde18015400 con 0x7fde20100930 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.552+0000 7fde177fe700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 0x7fde0c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:40.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.552+0000 7fde1dd9b700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 0x7fde0c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:40.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.552+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde180caf60 con 0x7fde20100930 2023-08-25T02:25:40.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.552+0000 7fde1dd9b700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 0x7fde0c0a2410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde20106010 tx=0x7fde08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:40.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.555+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fde18094da0 con 0x7fde20100930 2023-08-25T02:25:40.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168604 got 240518168606 for osd.8 2023-08-25T02:25:40.596 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:40.718 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561062 got 188978561064 for osd.6 2023-08-25T02:25:40.718 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:40.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.754+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fde20110a20 con 0x7fde20100930 2023-08-25T02:25:40.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.754+0000 7fde177fe700 1 -- 172.21.15.89:0/3136998349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde18024090 con 0x7fde20100930 2023-08-25T02:25:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:77309411385 2023-08-25T02:25:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 msgr2=0x7fde0c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 0x7fde0c0a2410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde20106010 tx=0x7fde08005fd0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 msgr2=0x7fde2019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fde180060b0 tx=0x7fde1800d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 shutdown_connections 2023-08-25T02:25:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde0c09ff50 0x7fde0c0a2410 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde20106930 0x7fde20196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde20100930 0x7fde2019c7b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 --2- 172.21.15.89:0/3136998349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde201012f0 0x7fde2019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:40.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 >> 172.21.15.89:0/3136998349 conn(0x7fde200fc420 msgr2=0x7fde200fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:40.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 shutdown_connections 2023-08-25T02:25:40.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:40.756+0000 7fde24fc0700 1 -- 172.21.15.89:0/3136998349 wait complete. 2023-08-25T02:25:41.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738434 for osd.0 2023-08-25T02:25:41.097 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:41.120 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939674 got 270582939675 for osd.9 2023-08-25T02:25:41.121 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:41.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574905 got 55834574908 for osd.1 2023-08-25T02:25:41.287 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:41.289 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:41.316 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:41 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3136998349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-08-25T02:25:41.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:41 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3136998349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-08-25T02:25:41.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:41 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3136998349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-08-25T02:25:41.813 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776148 got 292057776149 for osd.10 2023-08-25T02:25:41.814 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:42.291 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411385 for osd.2 2023-08-25T02:25:42.291 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:42.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:42 smithi089 ceph-mon[98732]: pgmap v269: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-08-25T02:25:42.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:42 smithi121 ceph-mon[100099]: pgmap v269: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-08-25T02:25:42.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:42 smithi122 ceph-mon[100457]: pgmap v269: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-08-25T02:25:43.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 -- 172.21.15.89:0/2403648901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c105960 msgr2=0x7fba3c105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 --2- 172.21.15.89:0/2403648901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c105960 0x7fba3c105d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fba30009990 tx=0x7fba30009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 -- 172.21.15.89:0/2403648901 shutdown_connections 2023-08-25T02:25:43.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 --2- 172.21.15.89:0/2403648901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 0x7fba3c10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 --2- 172.21.15.89:0/2403648901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c105960 0x7fba3c105d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 --2- 172.21.15.89:0/2403648901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0fee70 0x7fba3c0ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 -- 172.21.15.89:0/2403648901 >> 172.21.15.89:0/2403648901 conn(0x7fba3c075d90 msgr2=0x7fba3c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:43.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 -- 172.21.15.89:0/2403648901 shutdown_connections 2023-08-25T02:25:43.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.109+0000 7fba41741700 1 -- 172.21.15.89:0/2403648901 wait complete. 2023-08-25T02:25:43.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 Processor -- start 2023-08-25T02:25:43.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 -- start start 2023-08-25T02:25:43.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 0x7fba3c100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c105960 0x7fba3c1a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c100e80 con 0x7fba3c0fee70 2023-08-25T02:25:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c1a5350 con 0x7fba3c0ff830 2023-08-25T02:25:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba41741700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c1a54c0 con 0x7fba3c105960 2023-08-25T02:25:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47164/0 (socket says 172.21.15.89:47164) 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 -- 172.21.15.89:0/2184047907 learned_addr learned my addr 172.21.15.89:0/2184047907 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3a7fc700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 0x7fba3c100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 -- 172.21.15.89:0/2184047907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c105960 msgr2=0x7fba3c1a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c105960 0x7fba3c1a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 -- 172.21.15.89:0/2184047907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 msgr2=0x7fba3c100590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 0x7fba3c100590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.110+0000 7fba3affd700 1 -- 172.21.15.89:0/2184047907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba300095f0 con 0x7fba3c0fee70 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba3affd700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fba30005910 tx=0x7fba3000b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3001d070 con 0x7fba3c0fee70 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba3000fb80 con 0x7fba3c0fee70 2023-08-25T02:25:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c1a56e0 con 0x7fba3c0fee70 2023-08-25T02:25:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30017650 con 0x7fba3c0fee70 2023-08-25T02:25:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.111+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c1a5b50 con 0x7fba3c0fee70 2023-08-25T02:25:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.112+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c04e620 con 0x7fba3c0fee70 2023-08-25T02:25:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.113+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fba30022470 con 0x7fba3c0fee70 2023-08-25T02:25:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.113+0000 7fba23fff700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 0x7fba240a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.113+0000 7fba3a7fc700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 0x7fba240a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.113+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fba300cc740 con 0x7fba3c0fee70 2023-08-25T02:25:43.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.114+0000 7fba3a7fc700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 0x7fba240a2490 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fba3c101840 tx=0x7fba2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:43.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.117+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fba300966b0 con 0x7fba3c0fee70 2023-08-25T02:25:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.328+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fba3c110a20 con 0x7fba3c0fee70 2023-08-25T02:25:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.329+0000 7fba23fff700 1 -- 172.21.15.89:0/2184047907 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fba30099cd0 con 0x7fba3c0fee70 2023-08-25T02:25:43.330 INFO:teuthology.orchestra.run.smithi089.stdout:322122547219 2023-08-25T02:25:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 msgr2=0x7fba240a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 0x7fba240a2490 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fba3c101840 tx=0x7fba2c009260 comp rx=0 tx=0).stop 2023-08-25T02:25:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 msgr2=0x7fba3c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fba30005910 tx=0x7fba3000b610 comp rx=0 tx=0).stop 2023-08-25T02:25:43.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 shutdown_connections 2023-08-25T02:25:43.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c105960 0x7fba3c1a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0fee70 0x7fba3c100050 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2409ffd0 0x7fba240a2490 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 --2- 172.21.15.89:0/2184047907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba3c0ff830 0x7fba3c100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 >> 172.21.15.89:0/2184047907 conn(0x7fba3c075d90 msgr2=0x7fba3c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 shutdown_connections 2023-08-25T02:25:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.331+0000 7fba41741700 1 -- 172.21.15.89:0/2184047907 wait complete. 2023-08-25T02:25:43.428 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:43 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2184047907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-08-25T02:25:43.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 -- 172.21.15.89:0/662732352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c1001f0 msgr2=0x7fd90c10c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 --2- 172.21.15.89:0/662732352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c1001f0 0x7fd90c10c700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd90400b0a0 tx=0x7fd90400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 -- 172.21.15.89:0/662732352 shutdown_connections 2023-08-25T02:25:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 --2- 172.21.15.89:0/662732352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c1001f0 0x7fd90c10c700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 --2- 172.21.15.89:0/662732352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c105c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 --2- 172.21.15.89:0/662732352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c0ff830 0x7fd90c0ffcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 -- 172.21.15.89:0/662732352 >> 172.21.15.89:0/662732352 conn(0x7fd90c0fb320 msgr2=0x7fd90c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 -- 172.21.15.89:0/662732352 shutdown_connections 2023-08-25T02:25:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.451+0000 7fd912077700 1 -- 172.21.15.89:0/662732352 wait complete. 2023-08-25T02:25:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 Processor -- start 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 -- start start 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c1001f0 0x7fd90c10d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c10db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c1a4f80 con 0x7fd90c0ff830 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c1a50f0 con 0x7fd90c1001f0 2023-08-25T02:25:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd912077700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c1a52b0 con 0x7fd90c105830 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47178/0 (socket says 172.21.15.89:47178) 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.452+0000 7fd90b7fe700 1 -- 172.21.15.89:0/3878310273 learned_addr learned my addr 172.21.15.89:0/3878310273 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90affd700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c1001f0 0x7fd90c10d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90bfff700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c10db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 -- 172.21.15.89:0/3878310273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 msgr2=0x7fd90c10db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c10db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 -- 172.21.15.89:0/3878310273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c1001f0 msgr2=0x7fd90c10d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c1001f0 0x7fd90c10d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 -- 172.21.15.89:0/3878310273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009d00 con 0x7fd90c0ff830 2023-08-25T02:25:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90bfff700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd90b7fe700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00d950 tx=0x7fd8fc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc009b40 con 0x7fd90c0ff830 2023-08-25T02:25:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c1a5590 con 0x7fd90c0ff830 2023-08-25T02:25:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8fc009ca0 con 0x7fd90c0ff830 2023-08-25T02:25:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00f620 con 0x7fd90c0ff830 2023-08-25T02:25:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.453+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c1a5ae0 con 0x7fd90c0ff830 2023-08-25T02:25:43.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.454+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c04e620 con 0x7fd90c0ff830 2023-08-25T02:25:43.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.456+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd8fc010460 con 0x7fd90c0ff830 2023-08-25T02:25:43.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.456+0000 7fd908ff9700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 0x7fd8f40a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:43.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.457+0000 7fd90affd700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 0x7fd8f40a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:43.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.457+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd8fc0cb430 con 0x7fd90c0ff830 2023-08-25T02:25:43.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.457+0000 7fd90affd700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 0x7fd8f40a2410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd900005fd0 tx=0x7fd90000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:43.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.461+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd8fc095310 con 0x7fd90c0ff830 2023-08-25T02:25:43.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.662+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd90c110b30 con 0x7fd90c0ff830 2023-08-25T02:25:43.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.663+0000 7fd908ff9700 1 -- 172.21.15.89:0/3878310273 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd8fc098930 con 0x7fd90c0ff830 2023-08-25T02:25:43.665 INFO:teuthology.orchestra.run.smithi089.stdout:210453397541 2023-08-25T02:25:43.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 msgr2=0x7fd8f40a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 0x7fd8f40a2410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd900005fd0 tx=0x7fd90000a560 comp rx=0 tx=0).stop 2023-08-25T02:25:43.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 msgr2=0x7fd90c10d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:43.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00d950 tx=0x7fd8fc00dd10 comp rx=0 tx=0).stop 2023-08-25T02:25:43.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 shutdown_connections 2023-08-25T02:25:43.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd90c105830 0x7fd90c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0ff830 0x7fd90c10d0d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd8f409ff50 0x7fd8f40a2410 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 --2- 172.21.15.89:0/3878310273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd90c1001f0 0x7fd90c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 >> 172.21.15.89:0/3878310273 conn(0x7fd90c0fb320 msgr2=0x7fd90c0fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 shutdown_connections 2023-08-25T02:25:43.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:43.665+0000 7fd912077700 1 -- 172.21.15.89:0/3878310273 wait complete. 2023-08-25T02:25:43.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:43 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2184047907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-08-25T02:25:43.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:43 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2184047907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-08-25T02:25:43.820 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547217 got 322122547219 for osd.11 2023-08-25T02:25:43.820 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:44.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 -- 172.21.15.89:0/2040790547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105d50 msgr2=0x7f690810d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:44.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 --2- 172.21.15.89:0/2040790547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105d50 0x7f690810d760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f690000b0d0 tx=0x7f690000b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 -- 172.21.15.89:0/2040790547 shutdown_connections 2023-08-25T02:25:44.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 --2- 172.21.15.89:0/2040790547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105d50 0x7f690810d760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 --2- 172.21.15.89:0/2040790547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f6908068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 --2- 172.21.15.89:0/2040790547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105390 0x7f6908105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 -- 172.21.15.89:0/2040790547 >> 172.21.15.89:0/2040790547 conn(0x7f6908076920 msgr2=0x7f6908076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 -- 172.21.15.89:0/2040790547 shutdown_connections 2023-08-25T02:25:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.139+0000 7f690cf3c700 1 -- 172.21.15.89:0/2040790547 wait complete. 2023-08-25T02:25:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 Processor -- start 2023-08-25T02:25:44.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 -- start start 2023-08-25T02:25:44.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f690819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:44.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:44.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 0x7f6908196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:44.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908196fa0 con 0x7f6908105390 2023-08-25T02:25:44.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908197110 con 0x7f6908105d50 2023-08-25T02:25:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f690cf3c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69081972d0 con 0x7f69080688e0 2023-08-25T02:25:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.140+0000 7f6905d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:44.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47194/0 (socket says 172.21.15.89:47194) 2023-08-25T02:25:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 -- 172.21.15.89:0/472529279 learned_addr learned my addr 172.21.15.89:0/472529279 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f690659c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f690819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6906d9d700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 0x7f6908196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 -- 172.21.15.89:0/472529279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 msgr2=0x7f690819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:44.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f690819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 -- 172.21.15.89:0/472529279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 msgr2=0x7f6908196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 0x7f6908196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 -- 172.21.15.89:0/472529279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900009d00 con 0x7f6908105390 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f690659c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f690819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6906d9d700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 0x7f6908196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:44.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f6905d9b700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c920 tx=0x7f68f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8007b00 con 0x7f6908105390 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68f8004cf0 con 0x7f6908105390 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8005610 con 0x7f6908105390 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69081975b0 con 0x7f6908105390 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.141+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69081a38a0 con 0x7f6908105390 2023-08-25T02:25:44.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.142+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690804e620 con 0x7f6908105390 2023-08-25T02:25:44.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.143+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f68f8004750 con 0x7f6908105390 2023-08-25T02:25:44.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.144+0000 7f68f77fe700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 0x7f68f00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:44.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.144+0000 7f690659c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 0x7f68f00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:44.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.144+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f68f80cb1d0 con 0x7f6908105390 2023-08-25T02:25:44.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.144+0000 7f690659c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 0x7f68f00a2410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:44.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.148+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f68f80950b0 con 0x7f6908105390 2023-08-25T02:25:44.176 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397539 got 210453397541 for osd.7 2023-08-25T02:25:44.177 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:44.346 INFO:teuthology.orchestra.run.smithi089.stdout:107374182453 2023-08-25T02:25:44.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.345+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6908197e40 con 0x7f6908105390 2023-08-25T02:25:44.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.345+0000 7f68f77fe700 1 -- 172.21.15.89:0/472529279 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f68f801d090 con 0x7f6908105390 2023-08-25T02:25:44.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 msgr2=0x7f68f00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:44.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 0x7f68f00a2410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc005ee0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 msgr2=0x7f690819cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:44.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c920 tx=0x7f68f800cce0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 shutdown_connections 2023-08-25T02:25:44.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f68f009ff50 0x7f68f00a2410 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6908105d50 0x7f6908196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69080688e0 0x7f690819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 --2- 172.21.15.89:0/472529279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6908105390 0x7f690819cdf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:44.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 >> 172.21.15.89:0/472529279 conn(0x7f6908076920 msgr2=0x7f69080fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:44.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 shutdown_connections 2023-08-25T02:25:44.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:44.347+0000 7f690cf3c700 1 -- 172.21.15.89:0/472529279 wait complete. 2023-08-25T02:25:44.478 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:44 smithi089 ceph-mon[98732]: pgmap v270: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:44.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:44 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3878310273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-08-25T02:25:44.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:44 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/472529279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-08-25T02:25:44.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182453 for osd.3 2023-08-25T02:25:44.675 DEBUG:teuthology.parallel:result is None 2023-08-25T02:25:44.675 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-08-25T02:25:44.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph pg dump --format=json 2023-08-25T02:25:44.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:44 smithi121 ceph-mon[100099]: pgmap v270: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:44 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3878310273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-08-25T02:25:44.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:44 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/472529279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-08-25T02:25:44.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:44 smithi122 ceph-mon[100457]: pgmap v270: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:44.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:44 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3878310273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-08-25T02:25:44.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:44 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/472529279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-08-25T02:25:45.231 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 -- 172.21.15.89:0/761231175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508104730 msgr2=0x7f5508104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 --2- 172.21.15.89:0/761231175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508104730 0x7f5508104b10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009990 tx=0x7f54f8009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 -- 172.21.15.89:0/761231175 shutdown_connections 2023-08-25T02:25:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 --2- 172.21.15.89:0/761231175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508100220 0x7f550810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 --2- 172.21.15.89:0/761231175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508104730 0x7f5508104b10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 --2- 172.21.15.89:0/761231175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f55080ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 -- 172.21.15.89:0/761231175 >> 172.21.15.89:0/761231175 conn(0x7f5508076920 msgr2=0x7f5508076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:46.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 -- 172.21.15.89:0/761231175 shutdown_connections 2023-08-25T02:25:46.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.387+0000 7f550de8e700 1 -- 172.21.15.89:0/761231175 wait complete. 2023-08-25T02:25:46.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.388+0000 7f550de8e700 1 Processor -- start 2023-08-25T02:25:46.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.388+0000 7f550de8e700 1 -- start start 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.388+0000 7f550de8e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f550819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.388+0000 7f550de8e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.388+0000 7f550de8e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508104730 0x7f5508196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f550de8e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5508196f70 con 0x7f5508100220 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f550de8e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55081970e0 con 0x7f55080ff860 2023-08-25T02:25:46.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f550de8e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55081972a0 con 0x7f5508104730 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47206/0 (socket says 172.21.15.89:47206) 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 -- 172.21.15.89:0/3687544666 learned_addr learned my addr 172.21.15.89:0/3687544666 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f55077fe700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f550819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5507fff700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508104730 0x7f5508196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 -- 172.21.15.89:0/3687544666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508104730 msgr2=0x7f5508196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:46.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508104730 0x7f5508196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 -- 172.21.15.89:0/3687544666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 msgr2=0x7f550819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f550819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 -- 172.21.15.89:0/3687544666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f80095f0 con 0x7f5508100220 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f55077fe700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f550819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5506ffd700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b530 tx=0x7f54fc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc011830 con 0x7f5508100220 2023-08-25T02:25:46.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5508197580 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54fc011e50 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00f2e0 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.389+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55081a3870 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.390+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5508108d90 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.391+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f54fc0105f0 con 0x7f5508100220 2023-08-25T02:25:46.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.392+0000 7f5504ff9700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 0x7f54f00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:46.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.392+0000 7f55077fe700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 0x7f54f00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:46.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.392+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f54fc0cac00 con 0x7f5508100220 2023-08-25T02:25:46.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.392+0000 7f55077fe700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 0x7f54f00a2410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54f800b5c0 tx=0x7f54f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:46.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.396+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f54fc0d1080 con 0x7f5508100220 2023-08-25T02:25:46.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:46 smithi121 ceph-mon[100099]: pgmap v271: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:46.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:46 smithi122 ceph-mon[100457]: pgmap v271: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.578+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f550810f890 con 0x7f54f009ff50 2023-08-25T02:25:46.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:46 smithi089 ceph-mon[98732]: pgmap v271: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.588+0000 7f5504ff9700 1 -- 172.21.15.89:0/3687544666 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340879 (secure 0 0 0) 0x7f550810f890 con 0x7f54f009ff50 2023-08-25T02:25:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 msgr2=0x7f54f00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 0x7f54f00a2410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54f800b5c0 tx=0x7f54f8005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 msgr2=0x7f550819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b530 tx=0x7f54fc00b8f0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 shutdown_connections 2023-08-25T02:25:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f54f009ff50 0x7f54f00a2410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55080ff860 0x7f550819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5508104730 0x7f5508196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 --2- 172.21.15.89:0/3687544666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5508100220 0x7f550819cdc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:46.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 >> 172.21.15.89:0/3687544666 conn(0x7f5508076920 msgr2=0x7f5508068520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:46.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 shutdown_connections 2023-08-25T02:25:46.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:46.591+0000 7f550de8e700 1 -- 172.21.15.89:0/3687544666 wait complete. 2023-08-25T02:25:46.600 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-08-25T02:25:47.076 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2023-08-25T02:25:45.307717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":351,"num_read_kb":235,"num_write":306,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336216,"kb_used_data":12004,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605608,"statfs":{"total":1151940427776,"available":1151596142592,"internally_reserved":0,"allocated":12292096,"data_stored":6720410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003114"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214182+0000","last_change":"2023-08-25T02:25:04.226503+0000","last_active":"2023-08-25T02:25:05.214182+0000","last_peered":"2023-08-25T02:25:05.214182+0000","last_clean":"2023-08-25T02:25:05.214182+0000","last_became_active":"2023-08-25T02:25:04.226184+0000","last_became_peered":"2023-08-25T02:25:04.226184+0000","last_unstale":"2023-08-25T02:25:05.214182+0000","last_undegraded":"2023-08-25T02:25:05.214182+0000","last_fullsized":"2023-08-25T02:25:05.214182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:17:33.775196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938361+0000","last_change":"2023-08-25T02:25:04.236479+0000","last_active":"2023-08-25T02:25:05.938361+0000","last_peered":"2023-08-25T02:25:05.938361+0000","last_clean":"2023-08-25T02:25:05.938361+0000","last_became_active":"2023-08-25T02:25:04.235376+0000","last_became_peered":"2023-08-25T02:25:04.235376+0000","last_unstale":"2023-08-25T02:25:05.938361+0000","last_undegraded":"2023-08-25T02:25:05.938361+0000","last_fullsized":"2023-08-25T02:25:05.938361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:47.418998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938362+0000","last_change":"2023-08-25T02:25:04.235840+0000","last_active":"2023-08-25T02:25:05.938362+0000","last_peered":"2023-08-25T02:25:05.938362+0000","last_clean":"2023-08-25T02:25:05.938362+0000","last_became_active":"2023-08-25T02:25:04.235279+0000","last_became_peered":"2023-08-25T02:25:04.235279+0000","last_unstale":"2023-08-25T02:25:05.938362+0000","last_undegraded":"2023-08-25T02:25:05.938362+0000","last_fullsized":"2023-08-25T02:25:05.938362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:23.103019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474317+0000","last_change":"2023-08-25T02:25:04.237793+0000","last_active":"2023-08-25T02:25:06.474317+0000","last_peered":"2023-08-25T02:25:06.474317+0000","last_clean":"2023-08-25T02:25:06.474317+0000","last_became_active":"2023-08-25T02:25:04.237493+0000","last_became_peered":"2023-08-25T02:25:04.237493+0000","last_unstale":"2023-08-25T02:25:06.474317+0000","last_undegraded":"2023-08-25T02:25:06.474317+0000","last_fullsized":"2023-08-25T02:25:06.474317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:36:34.299231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657466+0000","last_change":"2023-08-25T02:25:04.221053+0000","last_active":"2023-08-25T02:25:06.657466+0000","last_peered":"2023-08-25T02:25:06.657466+0000","last_clean":"2023-08-25T02:25:06.657466+0000","last_became_active":"2023-08-25T02:25:04.220710+0000","last_became_peered":"2023-08-25T02:25:04.220710+0000","last_unstale":"2023-08-25T02:25:06.657466+0000","last_undegraded":"2023-08-25T02:25:06.657466+0000","last_fullsized":"2023-08-25T02:25:06.657466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:25:14.107245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214409+0000","last_change":"2023-08-25T02:25:04.234380+0000","last_active":"2023-08-25T02:25:05.214409+0000","last_peered":"2023-08-25T02:25:05.214409+0000","last_clean":"2023-08-25T02:25:05.214409+0000","last_became_active":"2023-08-25T02:25:04.233376+0000","last_became_peered":"2023-08-25T02:25:04.233376+0000","last_unstale":"2023-08-25T02:25:05.214409+0000","last_undegraded":"2023-08-25T02:25:05.214409+0000","last_fullsized":"2023-08-25T02:25:05.214409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:18:49.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473517+0000","last_change":"2023-08-25T02:25:04.228398+0000","last_active":"2023-08-25T02:25:06.473517+0000","last_peered":"2023-08-25T02:25:06.473517+0000","last_clean":"2023-08-25T02:25:06.473517+0000","last_became_active":"2023-08-25T02:25:04.227310+0000","last_became_peered":"2023-08-25T02:25:04.227310+0000","last_unstale":"2023-08-25T02:25:06.473517+0000","last_undegraded":"2023-08-25T02:25:06.473517+0000","last_fullsized":"2023-08-25T02:25:06.473517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:45:46.502139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213776+0000","last_change":"2023-08-25T02:25:04.235175+0000","last_active":"2023-08-25T02:25:05.213776+0000","last_peered":"2023-08-25T02:25:05.213776+0000","last_clean":"2023-08-25T02:25:05.213776+0000","last_became_active":"2023-08-25T02:25:04.234048+0000","last_became_peered":"2023-08-25T02:25:04.234048+0000","last_unstale":"2023-08-25T02:25:05.213776+0000","last_undegraded":"2023-08-25T02:25:05.213776+0000","last_fullsized":"2023-08-25T02:25:05.213776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:20.347389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438643+0000","last_change":"2023-08-25T02:25:04.226475+0000","last_active":"2023-08-25T02:25:06.438643+0000","last_peered":"2023-08-25T02:25:06.438643+0000","last_clean":"2023-08-25T02:25:06.438643+0000","last_became_active":"2023-08-25T02:25:04.226125+0000","last_became_peered":"2023-08-25T02:25:04.226125+0000","last_unstale":"2023-08-25T02:25:06.438643+0000","last_undegraded":"2023-08-25T02:25:06.438643+0000","last_fullsized":"2023-08-25T02:25:06.438643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:44.888916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214553+0000","last_change":"2023-08-25T02:25:04.234476+0000","last_active":"2023-08-25T02:25:05.214553+0000","last_peered":"2023-08-25T02:25:05.214553+0000","last_clean":"2023-08-25T02:25:05.214553+0000","last_became_active":"2023-08-25T02:25:04.234062+0000","last_became_peered":"2023-08-25T02:25:04.234062+0000","last_unstale":"2023-08-25T02:25:05.214553+0000","last_undegraded":"2023-08-25T02:25:05.214553+0000","last_fullsized":"2023-08-25T02:25:05.214553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:35:10.280467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473970+0000","last_change":"2023-08-25T02:25:04.233218+0000","last_active":"2023-08-25T02:25:06.473970+0000","last_peered":"2023-08-25T02:25:06.473970+0000","last_clean":"2023-08-25T02:25:06.473970+0000","last_became_active":"2023-08-25T02:25:04.232933+0000","last_became_peered":"2023-08-25T02:25:04.232933+0000","last_unstale":"2023-08-25T02:25:06.473970+0000","last_undegraded":"2023-08-25T02:25:06.473970+0000","last_fullsized":"2023-08-25T02:25:06.473970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:32:06.548589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474291+0000","last_change":"2023-08-25T02:25:04.238646+0000","last_active":"2023-08-25T02:25:06.474291+0000","last_peered":"2023-08-25T02:25:06.474291+0000","last_clean":"2023-08-25T02:25:06.474291+0000","last_became_active":"2023-08-25T02:25:04.238252+0000","last_became_peered":"2023-08-25T02:25:04.238252+0000","last_unstale":"2023-08-25T02:25:06.474291+0000","last_undegraded":"2023-08-25T02:25:06.474291+0000","last_fullsized":"2023-08-25T02:25:06.474291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:43:32.951632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473441+0000","last_change":"2023-08-25T02:25:04.236025+0000","last_active":"2023-08-25T02:25:06.473441+0000","last_peered":"2023-08-25T02:25:06.473441+0000","last_clean":"2023-08-25T02:25:06.473441+0000","last_became_active":"2023-08-25T02:25:04.235682+0000","last_became_peered":"2023-08-25T02:25:04.235682+0000","last_unstale":"2023-08-25T02:25:06.473441+0000","last_undegraded":"2023-08-25T02:25:06.473441+0000","last_fullsized":"2023-08-25T02:25:06.473441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:46.263361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938102+0000","last_change":"2023-08-25T02:25:04.236295+0000","last_active":"2023-08-25T02:25:05.938102+0000","last_peered":"2023-08-25T02:25:05.938102+0000","last_clean":"2023-08-25T02:25:05.938102+0000","last_became_active":"2023-08-25T02:25:04.235979+0000","last_became_peered":"2023-08-25T02:25:04.235979+0000","last_unstale":"2023-08-25T02:25:05.938102+0000","last_undegraded":"2023-08-25T02:25:05.938102+0000","last_fullsized":"2023-08-25T02:25:05.938102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:12:13.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938095+0000","last_change":"2023-08-25T02:25:04.234885+0000","last_active":"2023-08-25T02:25:05.938095+0000","last_peered":"2023-08-25T02:25:05.938095+0000","last_clean":"2023-08-25T02:25:05.938095+0000","last_became_active":"2023-08-25T02:25:04.234603+0000","last_became_peered":"2023-08-25T02:25:04.234603+0000","last_unstale":"2023-08-25T02:25:05.938095+0000","last_undegraded":"2023-08-25T02:25:05.938095+0000","last_fullsized":"2023-08-25T02:25:05.938095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:26.095674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438191+0000","last_change":"2023-08-25T02:25:04.230151+0000","last_active":"2023-08-25T02:25:06.438191+0000","last_peered":"2023-08-25T02:25:06.438191+0000","last_clean":"2023-08-25T02:25:06.438191+0000","last_became_active":"2023-08-25T02:25:04.227881+0000","last_became_peered":"2023-08-25T02:25:04.227881+0000","last_unstale":"2023-08-25T02:25:06.438191+0000","last_undegraded":"2023-08-25T02:25:06.438191+0000","last_fullsized":"2023-08-25T02:25:06.438191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:31.382761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214311+0000","last_change":"2023-08-25T02:25:04.230117+0000","last_active":"2023-08-25T02:25:05.214311+0000","last_peered":"2023-08-25T02:25:05.214311+0000","last_clean":"2023-08-25T02:25:05.214311+0000","last_became_active":"2023-08-25T02:25:04.229810+0000","last_became_peered":"2023-08-25T02:25:04.229810+0000","last_unstale":"2023-08-25T02:25:05.214311+0000","last_undegraded":"2023-08-25T02:25:05.214311+0000","last_fullsized":"2023-08-25T02:25:05.214311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:54:12.194687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473816+0000","last_change":"2023-08-25T02:25:04.235011+0000","last_active":"2023-08-25T02:25:06.473816+0000","last_peered":"2023-08-25T02:25:06.473816+0000","last_clean":"2023-08-25T02:25:06.473816+0000","last_became_active":"2023-08-25T02:25:04.234664+0000","last_became_peered":"2023-08-25T02:25:04.234664+0000","last_unstale":"2023-08-25T02:25:06.473816+0000","last_undegraded":"2023-08-25T02:25:06.473816+0000","last_fullsized":"2023-08-25T02:25:06.473816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:50:31.186711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473535+0000","last_change":"2023-08-25T02:25:04.229223+0000","last_active":"2023-08-25T02:25:06.473535+0000","last_peered":"2023-08-25T02:25:06.473535+0000","last_clean":"2023-08-25T02:25:06.473535+0000","last_became_active":"2023-08-25T02:25:04.227656+0000","last_became_peered":"2023-08-25T02:25:04.227656+0000","last_unstale":"2023-08-25T02:25:06.473535+0000","last_undegraded":"2023-08-25T02:25:06.473535+0000","last_fullsized":"2023-08-25T02:25:06.473535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:50:59.352227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'9","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640903+0000","last_change":"2023-08-25T02:25:03.613357+0000","last_active":"2023-08-25T02:25:08.640903+0000","last_peered":"2023-08-25T02:25:08.640903+0000","last_clean":"2023-08-25T02:25:08.640903+0000","last_became_active":"2023-08-25T02:25:03.612977+0000","last_became_peered":"2023-08-25T02:25:03.612977+0000","last_unstale":"2023-08-25T02:25:08.640903+0000","last_undegraded":"2023-08-25T02:25:08.640903+0000","last_fullsized":"2023-08-25T02:25:08.640903+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:28:01.586298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214090+0000","last_change":"2023-08-25T02:25:04.230544+0000","last_active":"2023-08-25T02:25:05.214090+0000","last_peered":"2023-08-25T02:25:05.214090+0000","last_clean":"2023-08-25T02:25:05.214090+0000","last_became_active":"2023-08-25T02:25:04.230226+0000","last_became_peered":"2023-08-25T02:25:04.230226+0000","last_unstale":"2023-08-25T02:25:05.214090+0000","last_undegraded":"2023-08-25T02:25:05.214090+0000","last_fullsized":"2023-08-25T02:25:05.214090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:41.647707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.830079+0000","last_change":"2023-08-25T02:25:03.617626+0000","last_active":"2023-08-25T02:25:05.830079+0000","last_peered":"2023-08-25T02:25:05.830079+0000","last_clean":"2023-08-25T02:25:05.830079+0000","last_became_active":"2023-08-25T02:25:03.617323+0000","last_became_peered":"2023-08-25T02:25:03.617323+0000","last_unstale":"2023-08-25T02:25:05.830079+0000","last_undegraded":"2023-08-25T02:25:05.830079+0000","last_fullsized":"2023-08-25T02:25:05.830079+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:05.360379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938528+0000","last_change":"2023-08-25T02:25:04.226808+0000","last_active":"2023-08-25T02:25:05.938528+0000","last_peered":"2023-08-25T02:25:05.938528+0000","last_clean":"2023-08-25T02:25:05.938528+0000","last_became_active":"2023-08-25T02:25:04.226407+0000","last_became_peered":"2023-08-25T02:25:04.226407+0000","last_unstale":"2023-08-25T02:25:05.938528+0000","last_undegraded":"2023-08-25T02:25:05.938528+0000","last_fullsized":"2023-08-25T02:25:05.938528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:26:23.478853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.639252+0000","last_change":"2023-08-25T02:25:03.611831+0000","last_active":"2023-08-25T02:25:08.639252+0000","last_peered":"2023-08-25T02:25:08.639252+0000","last_clean":"2023-08-25T02:25:08.639252+0000","last_became_active":"2023-08-25T02:25:03.611530+0000","last_became_peered":"2023-08-25T02:25:03.611530+0000","last_unstale":"2023-08-25T02:25:08.639252+0000","last_undegraded":"2023-08-25T02:25:08.639252+0000","last_fullsized":"2023-08-25T02:25:08.639252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:02:49.263476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438337+0000","last_change":"2023-08-25T02:25:04.236572+0000","last_active":"2023-08-25T02:25:06.438337+0000","last_peered":"2023-08-25T02:25:06.438337+0000","last_clean":"2023-08-25T02:25:06.438337+0000","last_became_active":"2023-08-25T02:25:04.236242+0000","last_became_peered":"2023-08-25T02:25:04.236242+0000","last_unstale":"2023-08-25T02:25:06.438337+0000","last_undegraded":"2023-08-25T02:25:06.438337+0000","last_fullsized":"2023-08-25T02:25:06.438337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:45:50.267973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657579+0000","last_change":"2023-08-25T02:25:03.616265+0000","last_active":"2023-08-25T02:25:06.657579+0000","last_peered":"2023-08-25T02:25:06.657579+0000","last_clean":"2023-08-25T02:25:06.657579+0000","last_became_active":"2023-08-25T02:25:03.615926+0000","last_became_peered":"2023-08-25T02:25:03.615926+0000","last_unstale":"2023-08-25T02:25:06.657579+0000","last_undegraded":"2023-08-25T02:25:06.657579+0000","last_fullsized":"2023-08-25T02:25:06.657579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:21:34.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157129+0000","last_change":"2023-08-25T02:25:04.232925+0000","last_active":"2023-08-25T02:25:06.157129+0000","last_peered":"2023-08-25T02:25:06.157129+0000","last_clean":"2023-08-25T02:25:06.157129+0000","last_became_active":"2023-08-25T02:25:04.232615+0000","last_became_peered":"2023-08-25T02:25:04.232615+0000","last_unstale":"2023-08-25T02:25:06.157129+0000","last_undegraded":"2023-08-25T02:25:06.157129+0000","last_fullsized":"2023-08-25T02:25:06.157129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:48:54.025387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:41.696058+0000","last_change":"2023-08-25T02:25:03.618534+0000","last_active":"2023-08-25T02:25:41.696058+0000","last_peered":"2023-08-25T02:25:41.696058+0000","last_clean":"2023-08-25T02:25:41.696058+0000","last_became_active":"2023-08-25T02:25:03.618284+0000","last_became_peered":"2023-08-25T02:25:03.618284+0000","last_unstale":"2023-08-25T02:25:41.696058+0000","last_undegraded":"2023-08-25T02:25:41.696058+0000","last_fullsized":"2023-08-25T02:25:41.696058+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:55:34.354817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657492+0000","last_change":"2023-08-25T02:25:04.225148+0000","last_active":"2023-08-25T02:25:06.657492+0000","last_peered":"2023-08-25T02:25:06.657492+0000","last_clean":"2023-08-25T02:25:06.657492+0000","last_became_active":"2023-08-25T02:25:04.224776+0000","last_became_peered":"2023-08-25T02:25:04.224776+0000","last_unstale":"2023-08-25T02:25:06.657492+0000","last_undegraded":"2023-08-25T02:25:06.657492+0000","last_fullsized":"2023-08-25T02:25:06.657492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:27.220482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641178+0000","last_change":"2023-08-25T02:25:03.610869+0000","last_active":"2023-08-25T02:25:08.641178+0000","last_peered":"2023-08-25T02:25:08.641178+0000","last_clean":"2023-08-25T02:25:08.641178+0000","last_became_active":"2023-08-25T02:25:03.610568+0000","last_became_peered":"2023-08-25T02:25:03.610568+0000","last_unstale":"2023-08-25T02:25:08.641178+0000","last_undegraded":"2023-08-25T02:25:08.641178+0000","last_fullsized":"2023-08-25T02:25:08.641178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:16.686802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157177+0000","last_change":"2023-08-25T02:25:04.230241+0000","last_active":"2023-08-25T02:25:06.157177+0000","last_peered":"2023-08-25T02:25:06.157177+0000","last_clean":"2023-08-25T02:25:06.157177+0000","last_became_active":"2023-08-25T02:25:04.229871+0000","last_became_peered":"2023-08-25T02:25:04.229871+0000","last_unstale":"2023-08-25T02:25:06.157177+0000","last_undegraded":"2023-08-25T02:25:06.157177+0000","last_fullsized":"2023-08-25T02:25:06.157177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:20:16.764817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641247+0000","last_change":"2023-08-25T02:25:03.610650+0000","last_active":"2023-08-25T02:25:08.641247+0000","last_peered":"2023-08-25T02:25:08.641247+0000","last_clean":"2023-08-25T02:25:08.641247+0000","last_became_active":"2023-08-25T02:25:03.609901+0000","last_became_peered":"2023-08-25T02:25:03.609901+0000","last_unstale":"2023-08-25T02:25:08.641247+0000","last_undegraded":"2023-08-25T02:25:08.641247+0000","last_fullsized":"2023-08-25T02:25:08.641247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:16.408779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213759+0000","last_change":"2023-08-25T02:25:04.222682+0000","last_active":"2023-08-25T02:25:05.213759+0000","last_peered":"2023-08-25T02:25:05.213759+0000","last_clean":"2023-08-25T02:25:05.213759+0000","last_became_active":"2023-08-25T02:25:04.221791+0000","last_became_peered":"2023-08-25T02:25:04.221791+0000","last_unstale":"2023-08-25T02:25:05.213759+0000","last_undegraded":"2023-08-25T02:25:05.213759+0000","last_fullsized":"2023-08-25T02:25:05.213759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:29:20.943592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156809+0000","last_change":"2023-08-25T02:25:03.610342+0000","last_active":"2023-08-25T02:25:06.156809+0000","last_peered":"2023-08-25T02:25:06.156809+0000","last_clean":"2023-08-25T02:25:06.156809+0000","last_became_active":"2023-08-25T02:25:03.609928+0000","last_became_peered":"2023-08-25T02:25:03.609928+0000","last_unstale":"2023-08-25T02:25:06.156809+0000","last_undegraded":"2023-08-25T02:25:06.156809+0000","last_fullsized":"2023-08-25T02:25:06.156809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:42.862108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473531+0000","last_change":"2023-08-25T02:25:04.234364+0000","last_active":"2023-08-25T02:25:06.473531+0000","last_peered":"2023-08-25T02:25:06.473531+0000","last_clean":"2023-08-25T02:25:06.473531+0000","last_became_active":"2023-08-25T02:25:04.234050+0000","last_became_peered":"2023-08-25T02:25:04.234050+0000","last_unstale":"2023-08-25T02:25:06.473531+0000","last_undegraded":"2023-08-25T02:25:06.473531+0000","last_fullsized":"2023-08-25T02:25:06.473531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:39:50.455964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:39.692255+0000","last_change":"2023-08-25T02:25:03.617016+0000","last_active":"2023-08-25T02:25:39.692255+0000","last_peered":"2023-08-25T02:25:39.692255+0000","last_clean":"2023-08-25T02:25:39.692255+0000","last_became_active":"2023-08-25T02:25:03.616670+0000","last_became_peered":"2023-08-25T02:25:03.616670+0000","last_unstale":"2023-08-25T02:25:39.692255+0000","last_undegraded":"2023-08-25T02:25:39.692255+0000","last_fullsized":"2023-08-25T02:25:39.692255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:42:03.302702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438625+0000","last_change":"2023-08-25T02:25:04.236009+0000","last_active":"2023-08-25T02:25:06.438625+0000","last_peered":"2023-08-25T02:25:06.438625+0000","last_clean":"2023-08-25T02:25:06.438625+0000","last_became_active":"2023-08-25T02:25:04.235722+0000","last_became_peered":"2023-08-25T02:25:04.235722+0000","last_unstale":"2023-08-25T02:25:06.438625+0000","last_undegraded":"2023-08-25T02:25:06.438625+0000","last_fullsized":"2023-08-25T02:25:06.438625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:42:44.656781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633826+0000","last_change":"2023-08-25T02:25:03.611060+0000","last_active":"2023-08-25T02:25:07.633826+0000","last_peered":"2023-08-25T02:25:07.633826+0000","last_clean":"2023-08-25T02:25:07.633826+0000","last_became_active":"2023-08-25T02:25:03.610715+0000","last_became_peered":"2023-08-25T02:25:03.610715+0000","last_unstale":"2023-08-25T02:25:07.633826+0000","last_undegraded":"2023-08-25T02:25:07.633826+0000","last_fullsized":"2023-08-25T02:25:07.633826+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:17.684716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938519+0000","last_change":"2023-08-25T02:25:04.225855+0000","last_active":"2023-08-25T02:25:05.938519+0000","last_peered":"2023-08-25T02:25:05.938519+0000","last_clean":"2023-08-25T02:25:05.938519+0000","last_became_active":"2023-08-25T02:25:04.225559+0000","last_became_peered":"2023-08-25T02:25:04.225559+0000","last_unstale":"2023-08-25T02:25:05.938519+0000","last_undegraded":"2023-08-25T02:25:05.938519+0000","last_fullsized":"2023-08-25T02:25:05.938519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:33:19.054268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632104+0000","last_change":"2023-08-25T02:25:03.617097+0000","last_active":"2023-08-25T02:25:07.632104+0000","last_peered":"2023-08-25T02:25:07.632104+0000","last_clean":"2023-08-25T02:25:07.632104+0000","last_became_active":"2023-08-25T02:25:03.616809+0000","last_became_peered":"2023-08-25T02:25:03.616809+0000","last_unstale":"2023-08-25T02:25:07.632104+0000","last_undegraded":"2023-08-25T02:25:07.632104+0000","last_fullsized":"2023-08-25T02:25:07.632104+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:11:24.908899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438483+0000","last_change":"2023-08-25T02:25:04.226384+0000","last_active":"2023-08-25T02:25:06.438483+0000","last_peered":"2023-08-25T02:25:06.438483+0000","last_clean":"2023-08-25T02:25:06.438483+0000","last_became_active":"2023-08-25T02:25:04.226047+0000","last_became_peered":"2023-08-25T02:25:04.226047+0000","last_unstale":"2023-08-25T02:25:06.438483+0000","last_undegraded":"2023-08-25T02:25:06.438483+0000","last_fullsized":"2023-08-25T02:25:06.438483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:01.485924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'8","reported_seq":88,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:40.693724+0000","last_change":"2023-08-25T02:25:03.609570+0000","last_active":"2023-08-25T02:25:40.693724+0000","last_peered":"2023-08-25T02:25:40.693724+0000","last_clean":"2023-08-25T02:25:40.693724+0000","last_became_active":"2023-08-25T02:25:03.609253+0000","last_became_peered":"2023-08-25T02:25:03.609253+0000","last_unstale":"2023-08-25T02:25:40.693724+0000","last_undegraded":"2023-08-25T02:25:40.693724+0000","last_fullsized":"2023-08-25T02:25:40.693724+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:14.775308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":66,"num_read_kb":34,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657588+0000","last_change":"2023-08-25T02:25:04.226667+0000","last_active":"2023-08-25T02:25:06.657588+0000","last_peered":"2023-08-25T02:25:06.657588+0000","last_clean":"2023-08-25T02:25:06.657588+0000","last_became_active":"2023-08-25T02:25:04.226332+0000","last_became_peered":"2023-08-25T02:25:04.226332+0000","last_unstale":"2023-08-25T02:25:06.657588+0000","last_undegraded":"2023-08-25T02:25:06.657588+0000","last_fullsized":"2023-08-25T02:25:06.657588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:51:57.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640547+0000","last_change":"2023-08-25T02:25:03.609419+0000","last_active":"2023-08-25T02:25:08.640547+0000","last_peered":"2023-08-25T02:25:08.640547+0000","last_clean":"2023-08-25T02:25:08.640547+0000","last_became_active":"2023-08-25T02:25:03.609101+0000","last_became_peered":"2023-08-25T02:25:03.609101+0000","last_unstale":"2023-08-25T02:25:08.640547+0000","last_undegraded":"2023-08-25T02:25:08.640547+0000","last_fullsized":"2023-08-25T02:25:08.640547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:15.692390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473753+0000","last_change":"2023-08-25T02:25:04.231351+0000","last_active":"2023-08-25T02:25:06.473753+0000","last_peered":"2023-08-25T02:25:06.473753+0000","last_clean":"2023-08-25T02:25:06.473753+0000","last_became_active":"2023-08-25T02:25:04.230974+0000","last_became_peered":"2023-08-25T02:25:04.230974+0000","last_unstale":"2023-08-25T02:25:06.473753+0000","last_undegraded":"2023-08-25T02:25:06.473753+0000","last_fullsized":"2023-08-25T02:25:06.473753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:12.882322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633217+0000","last_change":"2023-08-25T02:25:03.617889+0000","last_active":"2023-08-25T02:25:07.633217+0000","last_peered":"2023-08-25T02:25:07.633217+0000","last_clean":"2023-08-25T02:25:07.633217+0000","last_became_active":"2023-08-25T02:25:03.617576+0000","last_became_peered":"2023-08-25T02:25:03.617576+0000","last_unstale":"2023-08-25T02:25:07.633217+0000","last_undegraded":"2023-08-25T02:25:07.633217+0000","last_fullsized":"2023-08-25T02:25:07.633217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:38:22.627438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473924+0000","last_change":"2023-08-25T02:25:04.230135+0000","last_active":"2023-08-25T02:25:06.473924+0000","last_peered":"2023-08-25T02:25:06.473924+0000","last_clean":"2023-08-25T02:25:06.473924+0000","last_became_active":"2023-08-25T02:25:04.229592+0000","last_became_peered":"2023-08-25T02:25:04.229592+0000","last_unstale":"2023-08-25T02:25:06.473924+0000","last_undegraded":"2023-08-25T02:25:06.473924+0000","last_fullsized":"2023-08-25T02:25:06.473924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:28:47.602421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'3","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:40.693631+0000","last_change":"2023-08-25T02:25:03.610021+0000","last_active":"2023-08-25T02:25:40.693631+0000","last_peered":"2023-08-25T02:25:40.693631+0000","last_clean":"2023-08-25T02:25:40.693631+0000","last_became_active":"2023-08-25T02:25:03.609703+0000","last_became_peered":"2023-08-25T02:25:03.609703+0000","last_unstale":"2023-08-25T02:25:40.693631+0000","last_undegraded":"2023-08-25T02:25:40.693631+0000","last_fullsized":"2023-08-25T02:25:40.693631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:25.169266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473493+0000","last_change":"2023-08-25T02:25:04.234643+0000","last_active":"2023-08-25T02:25:06.473493+0000","last_peered":"2023-08-25T02:25:06.473493+0000","last_clean":"2023-08-25T02:25:06.473493+0000","last_became_active":"2023-08-25T02:25:04.234349+0000","last_became_peered":"2023-08-25T02:25:04.234349+0000","last_unstale":"2023-08-25T02:25:06.473493+0000","last_undegraded":"2023-08-25T02:25:06.473493+0000","last_fullsized":"2023-08-25T02:25:06.473493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:08:25.591726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640938+0000","last_change":"2023-08-25T02:25:03.613702+0000","last_active":"2023-08-25T02:25:08.640938+0000","last_peered":"2023-08-25T02:25:08.640938+0000","last_clean":"2023-08-25T02:25:08.640938+0000","last_became_active":"2023-08-25T02:25:03.613383+0000","last_became_peered":"2023-08-25T02:25:03.613383+0000","last_unstale":"2023-08-25T02:25:08.640938+0000","last_undegraded":"2023-08-25T02:25:08.640938+0000","last_fullsized":"2023-08-25T02:25:08.640938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:53.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156815+0000","last_change":"2023-08-25T02:25:04.227650+0000","last_active":"2023-08-25T02:25:06.156815+0000","last_peered":"2023-08-25T02:25:06.156815+0000","last_clean":"2023-08-25T02:25:06.156815+0000","last_became_active":"2023-08-25T02:25:04.227335+0000","last_became_peered":"2023-08-25T02:25:04.227335+0000","last_unstale":"2023-08-25T02:25:06.156815+0000","last_undegraded":"2023-08-25T02:25:06.156815+0000","last_fullsized":"2023-08-25T02:25:06.156815+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:10.086931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:09.643003+0000","last_change":"2023-08-25T02:25:03.616065+0000","last_active":"2023-08-25T02:25:09.643003+0000","last_peered":"2023-08-25T02:25:09.643003+0000","last_clean":"2023-08-25T02:25:09.643003+0000","last_became_active":"2023-08-25T02:25:03.615751+0000","last_became_peered":"2023-08-25T02:25:03.615751+0000","last_unstale":"2023-08-25T02:25:09.643003+0000","last_undegraded":"2023-08-25T02:25:09.643003+0000","last_fullsized":"2023-08-25T02:25:09.643003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:27:27.917233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214184+0000","last_change":"2023-08-25T02:25:04.228285+0000","last_active":"2023-08-25T02:25:05.214184+0000","last_peered":"2023-08-25T02:25:05.214184+0000","last_clean":"2023-08-25T02:25:05.214184+0000","last_became_active":"2023-08-25T02:25:04.228061+0000","last_became_peered":"2023-08-25T02:25:04.228061+0000","last_unstale":"2023-08-25T02:25:05.214184+0000","last_undegraded":"2023-08-25T02:25:05.214184+0000","last_fullsized":"2023-08-25T02:25:05.214184+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:40.994665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473497+0000","last_change":"2023-08-25T02:25:03.615712+0000","last_active":"2023-08-25T02:25:06.473497+0000","last_peered":"2023-08-25T02:25:06.473497+0000","last_clean":"2023-08-25T02:25:06.473497+0000","last_became_active":"2023-08-25T02:25:03.615412+0000","last_became_peered":"2023-08-25T02:25:03.615412+0000","last_unstale":"2023-08-25T02:25:06.473497+0000","last_undegraded":"2023-08-25T02:25:06.473497+0000","last_fullsized":"2023-08-25T02:25:06.473497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:19.940404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214408+0000","last_change":"2023-08-25T02:25:04.233485+0000","last_active":"2023-08-25T02:25:05.214408+0000","last_peered":"2023-08-25T02:25:05.214408+0000","last_clean":"2023-08-25T02:25:05.214408+0000","last_became_active":"2023-08-25T02:25:04.233205+0000","last_became_peered":"2023-08-25T02:25:04.233205+0000","last_unstale":"2023-08-25T02:25:05.214408+0000","last_undegraded":"2023-08-25T02:25:05.214408+0000","last_fullsized":"2023-08-25T02:25:05.214408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:12.219109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.413812+0000","last_change":"2023-08-25T02:25:03.613848+0000","last_active":"2023-08-25T02:25:07.413812+0000","last_peered":"2023-08-25T02:25:07.413812+0000","last_clean":"2023-08-25T02:25:07.413812+0000","last_became_active":"2023-08-25T02:25:03.613538+0000","last_became_peered":"2023-08-25T02:25:03.613538+0000","last_unstale":"2023-08-25T02:25:07.413812+0000","last_undegraded":"2023-08-25T02:25:07.413812+0000","last_fullsized":"2023-08-25T02:25:07.413812+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:52.883965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214475+0000","last_change":"2023-08-25T02:25:04.234167+0000","last_active":"2023-08-25T02:25:05.214475+0000","last_peered":"2023-08-25T02:25:05.214475+0000","last_clean":"2023-08-25T02:25:05.214475+0000","last_became_active":"2023-08-25T02:25:04.233779+0000","last_became_peered":"2023-08-25T02:25:04.233779+0000","last_unstale":"2023-08-25T02:25:05.214475+0000","last_undegraded":"2023-08-25T02:25:05.214475+0000","last_fullsized":"2023-08-25T02:25:05.214475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:19.007690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156909+0000","last_change":"2023-08-25T02:25:04.227759+0000","last_active":"2023-08-25T02:25:06.156909+0000","last_peered":"2023-08-25T02:25:06.156909+0000","last_clean":"2023-08-25T02:25:06.156909+0000","last_became_active":"2023-08-25T02:25:04.227166+0000","last_became_peered":"2023-08-25T02:25:04.227166+0000","last_unstale":"2023-08-25T02:25:06.156909+0000","last_undegraded":"2023-08-25T02:25:06.156909+0000","last_fullsized":"2023-08-25T02:25:06.156909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:58.916927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:39.691498+0000","last_change":"2023-08-25T02:25:03.613384+0000","last_active":"2023-08-25T02:25:39.691498+0000","last_peered":"2023-08-25T02:25:39.691498+0000","last_clean":"2023-08-25T02:25:39.691498+0000","last_became_active":"2023-08-25T02:25:03.613097+0000","last_became_peered":"2023-08-25T02:25:03.613097+0000","last_unstale":"2023-08-25T02:25:39.691498+0000","last_undegraded":"2023-08-25T02:25:39.691498+0000","last_fullsized":"2023-08-25T02:25:39.691498+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:55:46.368591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214227+0000","last_change":"2023-08-25T02:25:04.225009+0000","last_active":"2023-08-25T02:25:05.214227+0000","last_peered":"2023-08-25T02:25:05.214227+0000","last_clean":"2023-08-25T02:25:05.214227+0000","last_became_active":"2023-08-25T02:25:04.224678+0000","last_became_peered":"2023-08-25T02:25:04.224678+0000","last_unstale":"2023-08-25T02:25:05.214227+0000","last_undegraded":"2023-08-25T02:25:05.214227+0000","last_fullsized":"2023-08-25T02:25:05.214227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:45:44.498691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'3","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:09.642186+0000","last_change":"2023-08-25T02:25:03.607620+0000","last_active":"2023-08-25T02:25:09.642186+0000","last_peered":"2023-08-25T02:25:09.642186+0000","last_clean":"2023-08-25T02:25:09.642186+0000","last_became_active":"2023-08-25T02:25:03.607329+0000","last_became_peered":"2023-08-25T02:25:03.607329+0000","last_unstale":"2023-08-25T02:25:09.642186+0000","last_undegraded":"2023-08-25T02:25:09.642186+0000","last_fullsized":"2023-08-25T02:25:09.642186+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:21.994627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473471+0000","last_change":"2023-08-25T02:24:05.131624+0000","last_active":"2023-08-25T02:25:06.473471+0000","last_peered":"2023-08-25T02:25:06.473471+0000","last_clean":"2023-08-25T02:25:06.473471+0000","last_became_active":"2023-08-25T02:24:05.131084+0000","last_became_peered":"2023-08-25T02:24:05.131084+0000","last_unstale":"2023-08-25T02:25:06.473471+0000","last_undegraded":"2023-08-25T02:25:06.473471+0000","last_fullsized":"2023-08-25T02:25:06.473471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_clean_scrub_stamp":"2023-08-25T02:21:10.049896+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:44.618723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473109+0000","last_change":"2023-08-25T02:25:04.231797+0000","last_active":"2023-08-25T02:25:06.473109+0000","last_peered":"2023-08-25T02:25:06.473109+0000","last_clean":"2023-08-25T02:25:06.473109+0000","last_became_active":"2023-08-25T02:25:04.231520+0000","last_became_peered":"2023-08-25T02:25:04.231520+0000","last_unstale":"2023-08-25T02:25:06.473109+0000","last_undegraded":"2023-08-25T02:25:06.473109+0000","last_fullsized":"2023-08-25T02:25:06.473109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:30:43.262427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.639630+0000","last_change":"2023-08-25T02:25:03.611517+0000","last_active":"2023-08-25T02:25:08.639630+0000","last_peered":"2023-08-25T02:25:08.639630+0000","last_clean":"2023-08-25T02:25:08.639630+0000","last_became_active":"2023-08-25T02:25:03.611205+0000","last_became_peered":"2023-08-25T02:25:03.611205+0000","last_unstale":"2023-08-25T02:25:08.639630+0000","last_undegraded":"2023-08-25T02:25:08.639630+0000","last_fullsized":"2023-08-25T02:25:08.639630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:04:13.071834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.631473+0000","last_change":"2023-08-25T02:25:03.613795+0000","last_active":"2023-08-25T02:25:07.631473+0000","last_peered":"2023-08-25T02:25:07.631473+0000","last_clean":"2023-08-25T02:25:07.631473+0000","last_became_active":"2023-08-25T02:25:03.613410+0000","last_became_peered":"2023-08-25T02:25:03.613410+0000","last_unstale":"2023-08-25T02:25:07.631473+0000","last_undegraded":"2023-08-25T02:25:07.631473+0000","last_fullsized":"2023-08-25T02:25:07.631473+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:47:37.646320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473432+0000","last_change":"2023-08-25T02:25:04.232186+0000","last_active":"2023-08-25T02:25:06.473432+0000","last_peered":"2023-08-25T02:25:06.473432+0000","last_clean":"2023-08-25T02:25:06.473432+0000","last_became_active":"2023-08-25T02:25:04.231853+0000","last_became_peered":"2023-08-25T02:25:04.231853+0000","last_unstale":"2023-08-25T02:25:06.473432+0000","last_undegraded":"2023-08-25T02:25:06.473432+0000","last_fullsized":"2023-08-25T02:25:06.473432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:07:47.816234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641720+0000","last_change":"2023-08-25T02:25:03.615541+0000","last_active":"2023-08-25T02:25:08.641720+0000","last_peered":"2023-08-25T02:25:08.641720+0000","last_clean":"2023-08-25T02:25:08.641720+0000","last_became_active":"2023-08-25T02:25:03.615196+0000","last_became_peered":"2023-08-25T02:25:03.615196+0000","last_unstale":"2023-08-25T02:25:08.641720+0000","last_undegraded":"2023-08-25T02:25:08.641720+0000","last_fullsized":"2023-08-25T02:25:08.641720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:46:14.320885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214288+0000","last_change":"2023-08-25T02:25:04.227688+0000","last_active":"2023-08-25T02:25:05.214288+0000","last_peered":"2023-08-25T02:25:05.214288+0000","last_clean":"2023-08-25T02:25:05.214288+0000","last_became_active":"2023-08-25T02:25:04.227394+0000","last_became_peered":"2023-08-25T02:25:04.227394+0000","last_unstale":"2023-08-25T02:25:05.214288+0000","last_undegraded":"2023-08-25T02:25:05.214288+0000","last_fullsized":"2023-08-25T02:25:05.214288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:53:57.237749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633564+0000","last_change":"2023-08-25T02:25:03.608500+0000","last_active":"2023-08-25T02:25:07.633564+0000","last_peered":"2023-08-25T02:25:07.633564+0000","last_clean":"2023-08-25T02:25:07.633564+0000","last_became_active":"2023-08-25T02:25:03.608172+0000","last_became_peered":"2023-08-25T02:25:03.608172+0000","last_unstale":"2023-08-25T02:25:07.633564+0000","last_undegraded":"2023-08-25T02:25:07.633564+0000","last_fullsized":"2023-08-25T02:25:07.633564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:18:45.063308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938372+0000","last_change":"2023-08-25T02:25:04.226768+0000","last_active":"2023-08-25T02:25:05.938372+0000","last_peered":"2023-08-25T02:25:05.938372+0000","last_clean":"2023-08-25T02:25:05.938372+0000","last_became_active":"2023-08-25T02:25:04.226459+0000","last_became_peered":"2023-08-25T02:25:04.226459+0000","last_unstale":"2023-08-25T02:25:05.938372+0000","last_undegraded":"2023-08-25T02:25:05.938372+0000","last_fullsized":"2023-08-25T02:25:05.938372+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:04:52.374035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633766+0000","last_change":"2023-08-25T02:25:03.612306+0000","last_active":"2023-08-25T02:25:07.633766+0000","last_peered":"2023-08-25T02:25:07.633766+0000","last_clean":"2023-08-25T02:25:07.633766+0000","last_became_active":"2023-08-25T02:25:03.611922+0000","last_became_peered":"2023-08-25T02:25:03.611922+0000","last_unstale":"2023-08-25T02:25:07.633766+0000","last_undegraded":"2023-08-25T02:25:07.633766+0000","last_fullsized":"2023-08-25T02:25:07.633766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:14:09.728422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657329+0000","last_change":"2023-08-25T02:25:04.235032+0000","last_active":"2023-08-25T02:25:06.657329+0000","last_peered":"2023-08-25T02:25:06.657329+0000","last_clean":"2023-08-25T02:25:06.657329+0000","last_became_active":"2023-08-25T02:25:04.234626+0000","last_became_peered":"2023-08-25T02:25:04.234626+0000","last_unstale":"2023-08-25T02:25:06.657329+0000","last_undegraded":"2023-08-25T02:25:06.657329+0000","last_fullsized":"2023-08-25T02:25:06.657329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:00.849934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632365+0000","last_change":"2023-08-25T02:25:03.608869+0000","last_active":"2023-08-25T02:25:07.632365+0000","last_peered":"2023-08-25T02:25:07.632365+0000","last_clean":"2023-08-25T02:25:07.632365+0000","last_became_active":"2023-08-25T02:25:03.608530+0000","last_became_peered":"2023-08-25T02:25:03.608530+0000","last_unstale":"2023-08-25T02:25:07.632365+0000","last_undegraded":"2023-08-25T02:25:07.632365+0000","last_fullsized":"2023-08-25T02:25:07.632365+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:51:39.415744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213929+0000","last_change":"2023-08-25T02:25:04.231830+0000","last_active":"2023-08-25T02:25:05.213929+0000","last_peered":"2023-08-25T02:25:05.213929+0000","last_clean":"2023-08-25T02:25:05.213929+0000","last_became_active":"2023-08-25T02:25:04.231530+0000","last_became_peered":"2023-08-25T02:25:04.231530+0000","last_unstale":"2023-08-25T02:25:05.213929+0000","last_undegraded":"2023-08-25T02:25:05.213929+0000","last_fullsized":"2023-08-25T02:25:05.213929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:34:31.230104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473570+0000","last_change":"2023-08-25T02:25:03.610625+0000","last_active":"2023-08-25T02:25:06.473570+0000","last_peered":"2023-08-25T02:25:06.473570+0000","last_clean":"2023-08-25T02:25:06.473570+0000","last_became_active":"2023-08-25T02:25:03.610305+0000","last_became_peered":"2023-08-25T02:25:03.610305+0000","last_unstale":"2023-08-25T02:25:06.473570+0000","last_undegraded":"2023-08-25T02:25:06.473570+0000","last_fullsized":"2023-08-25T02:25:06.473570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:28:44.209156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214041+0000","last_change":"2023-08-25T02:25:04.234533+0000","last_active":"2023-08-25T02:25:05.214041+0000","last_peered":"2023-08-25T02:25:05.214041+0000","last_clean":"2023-08-25T02:25:05.214041+0000","last_became_active":"2023-08-25T02:25:04.234167+0000","last_became_peered":"2023-08-25T02:25:04.234167+0000","last_unstale":"2023-08-25T02:25:05.214041+0000","last_undegraded":"2023-08-25T02:25:05.214041+0000","last_fullsized":"2023-08-25T02:25:05.214041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:23:55.422360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633449+0000","last_change":"2023-08-25T02:25:03.610139+0000","last_active":"2023-08-25T02:25:07.633449+0000","last_peered":"2023-08-25T02:25:07.633449+0000","last_clean":"2023-08-25T02:25:07.633449+0000","last_became_active":"2023-08-25T02:25:03.609811+0000","last_became_peered":"2023-08-25T02:25:03.609811+0000","last_unstale":"2023-08-25T02:25:07.633449+0000","last_undegraded":"2023-08-25T02:25:07.633449+0000","last_fullsized":"2023-08-25T02:25:07.633449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:01.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474201+0000","last_change":"2023-08-25T02:25:04.233179+0000","last_active":"2023-08-25T02:25:06.474201+0000","last_peered":"2023-08-25T02:25:06.474201+0000","last_clean":"2023-08-25T02:25:06.474201+0000","last_became_active":"2023-08-25T02:25:04.232874+0000","last_became_peered":"2023-08-25T02:25:04.232874+0000","last_unstale":"2023-08-25T02:25:06.474201+0000","last_undegraded":"2023-08-25T02:25:06.474201+0000","last_fullsized":"2023-08-25T02:25:06.474201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:54.317603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641063+0000","last_change":"2023-08-25T02:25:03.619258+0000","last_active":"2023-08-25T02:25:08.641063+0000","last_peered":"2023-08-25T02:25:08.641063+0000","last_clean":"2023-08-25T02:25:08.641063+0000","last_became_active":"2023-08-25T02:25:03.618954+0000","last_became_peered":"2023-08-25T02:25:03.618954+0000","last_unstale":"2023-08-25T02:25:08.641063+0000","last_undegraded":"2023-08-25T02:25:08.641063+0000","last_fullsized":"2023-08-25T02:25:08.641063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:39:48.139135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474059+0000","last_change":"2023-08-25T02:25:04.232620+0000","last_active":"2023-08-25T02:25:06.474059+0000","last_peered":"2023-08-25T02:25:06.474059+0000","last_clean":"2023-08-25T02:25:06.474059+0000","last_became_active":"2023-08-25T02:25:04.232314+0000","last_became_peered":"2023-08-25T02:25:04.232314+0000","last_unstale":"2023-08-25T02:25:06.474059+0000","last_undegraded":"2023-08-25T02:25:06.474059+0000","last_fullsized":"2023-08-25T02:25:06.474059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:39:46.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157121+0000","last_change":"2023-08-25T02:25:03.617794+0000","last_active":"2023-08-25T02:25:06.157121+0000","last_peered":"2023-08-25T02:25:06.157121+0000","last_clean":"2023-08-25T02:25:06.157121+0000","last_became_active":"2023-08-25T02:25:03.617521+0000","last_became_peered":"2023-08-25T02:25:03.617521+0000","last_unstale":"2023-08-25T02:25:06.157121+0000","last_undegraded":"2023-08-25T02:25:06.157121+0000","last_fullsized":"2023-08-25T02:25:06.157121+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:35:22.049419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474136+0000","last_change":"2023-08-25T02:25:04.229796+0000","last_active":"2023-08-25T02:25:06.474136+0000","last_peered":"2023-08-25T02:25:06.474136+0000","last_clean":"2023-08-25T02:25:06.474136+0000","last_became_active":"2023-08-25T02:25:04.227073+0000","last_became_peered":"2023-08-25T02:25:04.227073+0000","last_unstale":"2023-08-25T02:25:06.474136+0000","last_undegraded":"2023-08-25T02:25:06.474136+0000","last_fullsized":"2023-08-25T02:25:06.474136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:06:43.826591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473306+0000","last_change":"2023-08-25T02:25:03.612702+0000","last_active":"2023-08-25T02:25:06.473306+0000","last_peered":"2023-08-25T02:25:06.473306+0000","last_clean":"2023-08-25T02:25:06.473306+0000","last_became_active":"2023-08-25T02:25:03.612482+0000","last_became_peered":"2023-08-25T02:25:03.612482+0000","last_unstale":"2023-08-25T02:25:06.473306+0000","last_undegraded":"2023-08-25T02:25:06.473306+0000","last_fullsized":"2023-08-25T02:25:06.473306+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:02:34.423511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474025+0000","last_change":"2023-08-25T02:25:04.229892+0000","last_active":"2023-08-25T02:25:06.474025+0000","last_peered":"2023-08-25T02:25:06.474025+0000","last_clean":"2023-08-25T02:25:06.474025+0000","last_became_active":"2023-08-25T02:25:04.229581+0000","last_became_peered":"2023-08-25T02:25:04.229581+0000","last_unstale":"2023-08-25T02:25:06.474025+0000","last_undegraded":"2023-08-25T02:25:06.474025+0000","last_fullsized":"2023-08-25T02:25:06.474025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:15:54.252164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214023+0000","last_change":"2023-08-25T02:25:04.231992+0000","last_active":"2023-08-25T02:25:05.214023+0000","last_peered":"2023-08-25T02:25:05.214023+0000","last_clean":"2023-08-25T02:25:05.214023+0000","last_became_active":"2023-08-25T02:25:04.231676+0000","last_became_peered":"2023-08-25T02:25:04.231676+0000","last_unstale":"2023-08-25T02:25:05.214023+0000","last_undegraded":"2023-08-25T02:25:05.214023+0000","last_fullsized":"2023-08-25T02:25:05.214023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:47:35.254078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473483+0000","last_change":"2023-08-25T02:25:04.227123+0000","last_active":"2023-08-25T02:25:06.473483+0000","last_peered":"2023-08-25T02:25:06.473483+0000","last_clean":"2023-08-25T02:25:06.473483+0000","last_became_active":"2023-08-25T02:25:04.226828+0000","last_became_peered":"2023-08-25T02:25:04.226828+0000","last_unstale":"2023-08-25T02:25:06.473483+0000","last_undegraded":"2023-08-25T02:25:06.473483+0000","last_fullsized":"2023-08-25T02:25:06.473483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:22.044584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473472+0000","last_change":"2023-08-25T02:25:04.232635+0000","last_active":"2023-08-25T02:25:06.473472+0000","last_peered":"2023-08-25T02:25:06.473472+0000","last_clean":"2023-08-25T02:25:06.473472+0000","last_became_active":"2023-08-25T02:25:04.232353+0000","last_became_peered":"2023-08-25T02:25:04.232353+0000","last_unstale":"2023-08-25T02:25:06.473472+0000","last_undegraded":"2023-08-25T02:25:06.473472+0000","last_fullsized":"2023-08-25T02:25:06.473472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:14:24.041499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938254+0000","last_change":"2023-08-25T02:25:04.226977+0000","last_active":"2023-08-25T02:25:05.938254+0000","last_peered":"2023-08-25T02:25:05.938254+0000","last_clean":"2023-08-25T02:25:05.938254+0000","last_became_active":"2023-08-25T02:25:04.226579+0000","last_became_peered":"2023-08-25T02:25:04.226579+0000","last_unstale":"2023-08-25T02:25:05.938254+0000","last_undegraded":"2023-08-25T02:25:05.938254+0000","last_fullsized":"2023-08-25T02:25:05.938254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:00.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473432+0000","last_change":"2023-08-25T02:25:04.225050+0000","last_active":"2023-08-25T02:25:06.473432+0000","last_peered":"2023-08-25T02:25:06.473432+0000","last_clean":"2023-08-25T02:25:06.473432+0000","last_became_active":"2023-08-25T02:25:04.224720+0000","last_became_peered":"2023-08-25T02:25:04.224720+0000","last_unstale":"2023-08-25T02:25:06.473432+0000","last_undegraded":"2023-08-25T02:25:06.473432+0000","last_fullsized":"2023-08-25T02:25:06.473432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:05:00.583742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214172+0000","last_change":"2023-08-25T02:25:04.228996+0000","last_active":"2023-08-25T02:25:05.214172+0000","last_peered":"2023-08-25T02:25:05.214172+0000","last_clean":"2023-08-25T02:25:05.214172+0000","last_became_active":"2023-08-25T02:25:04.228526+0000","last_became_peered":"2023-08-25T02:25:04.228526+0000","last_unstale":"2023-08-25T02:25:05.214172+0000","last_undegraded":"2023-08-25T02:25:05.214172+0000","last_fullsized":"2023-08-25T02:25:05.214172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:08:48.971024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473735+0000","last_change":"2023-08-25T02:25:04.234921+0000","last_active":"2023-08-25T02:25:06.473735+0000","last_peered":"2023-08-25T02:25:06.473735+0000","last_clean":"2023-08-25T02:25:06.473735+0000","last_became_active":"2023-08-25T02:25:04.234588+0000","last_became_peered":"2023-08-25T02:25:04.234588+0000","last_unstale":"2023-08-25T02:25:06.473735+0000","last_undegraded":"2023-08-25T02:25:06.473735+0000","last_fullsized":"2023-08-25T02:25:06.473735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:21:48.506843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214180+0000","last_change":"2023-08-25T02:25:04.227310+0000","last_active":"2023-08-25T02:25:05.214180+0000","last_peered":"2023-08-25T02:25:05.214180+0000","last_clean":"2023-08-25T02:25:05.214180+0000","last_became_active":"2023-08-25T02:25:04.226940+0000","last_became_peered":"2023-08-25T02:25:04.226940+0000","last_unstale":"2023-08-25T02:25:05.214180+0000","last_undegraded":"2023-08-25T02:25:05.214180+0000","last_fullsized":"2023-08-25T02:25:05.214180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:21:22.991356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156816+0000","last_change":"2023-08-25T02:25:04.230835+0000","last_active":"2023-08-25T02:25:06.156816+0000","last_peered":"2023-08-25T02:25:06.156816+0000","last_clean":"2023-08-25T02:25:06.156816+0000","last_became_active":"2023-08-25T02:25:04.230534+0000","last_became_peered":"2023-08-25T02:25:04.230534+0000","last_unstale":"2023-08-25T02:25:06.156816+0000","last_undegraded":"2023-08-25T02:25:06.156816+0000","last_fullsized":"2023-08-25T02:25:06.156816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:09.527955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213887+0000","last_change":"2023-08-25T02:25:04.224250+0000","last_active":"2023-08-25T02:25:05.213887+0000","last_peered":"2023-08-25T02:25:05.213887+0000","last_clean":"2023-08-25T02:25:05.213887+0000","last_became_active":"2023-08-25T02:25:04.223591+0000","last_became_peered":"2023-08-25T02:25:04.223591+0000","last_unstale":"2023-08-25T02:25:05.213887+0000","last_undegraded":"2023-08-25T02:25:05.213887+0000","last_fullsized":"2023-08-25T02:25:05.213887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:26:52.465008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438356+0000","last_change":"2023-08-25T02:25:04.229455+0000","last_active":"2023-08-25T02:25:06.438356+0000","last_peered":"2023-08-25T02:25:06.438356+0000","last_clean":"2023-08-25T02:25:06.438356+0000","last_became_active":"2023-08-25T02:25:04.228114+0000","last_became_peered":"2023-08-25T02:25:04.228114+0000","last_unstale":"2023-08-25T02:25:06.438356+0000","last_undegraded":"2023-08-25T02:25:06.438356+0000","last_fullsized":"2023-08-25T02:25:06.438356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:12:29.745750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473576+0000","last_change":"2023-08-25T02:25:04.226625+0000","last_active":"2023-08-25T02:25:06.473576+0000","last_peered":"2023-08-25T02:25:06.473576+0000","last_clean":"2023-08-25T02:25:06.473576+0000","last_became_active":"2023-08-25T02:25:04.226253+0000","last_became_peered":"2023-08-25T02:25:04.226253+0000","last_unstale":"2023-08-25T02:25:06.473576+0000","last_undegraded":"2023-08-25T02:25:06.473576+0000","last_fullsized":"2023-08-25T02:25:06.473576+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:59:23.362440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214063+0000","last_change":"2023-08-25T02:25:04.223776+0000","last_active":"2023-08-25T02:25:05.214063+0000","last_peered":"2023-08-25T02:25:05.214063+0000","last_clean":"2023-08-25T02:25:05.214063+0000","last_became_active":"2023-08-25T02:25:04.223597+0000","last_became_peered":"2023-08-25T02:25:04.223597+0000","last_unstale":"2023-08-25T02:25:05.214063+0000","last_undegraded":"2023-08-25T02:25:05.214063+0000","last_fullsized":"2023-08-25T02:25:05.214063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:26:39.163992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":215,"num_read_kb":116,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":15666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547220,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":411769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.67800000000000005}]},{"osd":2,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.63}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.84399999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.69099999999999995}]}]},{"osd":10,"up_from":68,"seq":292057776151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1004745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.88},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.71499999999999997}]},{"osd":3,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.68400000000000005}]},{"osd":4,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":1.012},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.91500000000000004}]},{"osd":6,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.96699999999999997}]},{"osd":9,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.93400000000000005}]},{"osd":11,"last update":"Fri Aug 25 02:25:26 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.98999999999999999}]}]},{"osd":9,"up_from":63,"seq":270582939676,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":412005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.28299999999999997}]},{"osd":1,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.495},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.35599999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58299999999999996}]},{"osd":8,"last update":"Fri Aug 25 02:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.52500000000000002}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.55200000000000005}]}]},{"osd":8,"up_from":56,"seq":240518168608,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":411925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.498}]},{"osd":1,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.86499999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66600000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.433,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.107,"15min":1.107},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.627,"15min":0.626},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78100000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.77000000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74099999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.71299999999999997}]},{"osd":7,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.74299999999999999,"15min":0.72999999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.53100000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.0489999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.82699999999999996}]},{"osd":9,"last update":"Fri Aug 25 02:25:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.745,"15min":0.73099999999999998},"min":{"1min":0.58699999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.022,"5min":0.78900000000000003,"15min":0.75},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.9750000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.85599999999999998}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.78300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":7,"up_from":49,"seq":210453397541,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":1003347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.497}]},{"osd":1,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42599999999999999,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.432},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60199999999999998,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.73599999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.65200000000000002,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92100000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70499999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.67100000000000004}]},{"osd":8,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.71699999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.72299999999999998,"15min":0.70899999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.89300000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.90600000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561065,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":860160,"data_stored":411277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.28999999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57299999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.34599999999999997}]},{"osd":1,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68300000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.627}]},{"osd":3,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":3.5659999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.45200000000000001},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.76200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.628,"15min":0.628},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79700000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83099999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.61799999999999999}]},{"osd":7,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.46800000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66800000000000004}]},{"osd":8,"last update":"Fri Aug 25 02:25:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.52900000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.66000000000000003}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59999999999999998}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.63900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":412526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.55800000000000005,"15min":0.504},"min":{"1min":0.42699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.53000000000000003,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.625}]},{"osd":3,"last update":"Fri Aug 25 02:25:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78000000000000003,"5min":0.997,"15min":0.997},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53400000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67000000000000004,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.747}]},{"osd":4,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.39300000000000002},"min":{"1min":0.33600000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.45900000000000002,"15min":0.41999999999999998},"min":{"1min":0.36499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.52400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60099999999999998,"15min":0.57899999999999996},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73099999999999998}]},{"osd":6,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.64900000000000002,"15min":0.61899999999999999},"min":{"1min":0.54100000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.51000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77600000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.439}]},{"osd":8,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.72499999999999998,"15min":0.70399999999999996},"min":{"1min":0.54400000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.55100000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.876},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":412665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.434,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.42999999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72999999999999998,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.50600000000000001},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63700000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.80500000000000005}]},{"osd":3,"last update":"Fri Aug 25 02:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.33900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.79300000000000004}]},{"osd":4,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.436,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70299999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.42799999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57399999999999995}]},{"osd":6,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69699999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.754,"15min":0.754},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83899999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78500000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.82199999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:24:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64200000000000002}]},{"osd":10,"last update":"Fri Aug 25 02:25:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.77800000000000002}]},{"osd":11,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.748},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.878}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":414149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51600000000000001,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.496},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50600000000000001,"15min":0.46700000000000003},"min":{"1min":0.377,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79400000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.54000000000000004},"min":{"1min":0.46100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77400000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80800000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Aug 25 02:25:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.52200000000000002},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80600000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58199999999999996,"15min":0.54200000000000004},"min":{"1min":0.504,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81000000000000005,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Aug 25 02:24:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76800000000000002,"5min":0.871,"15min":0.871},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.44800000000000001,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65000000000000002}]},{"osd":6,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.66100000000000003,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.746}]},{"osd":9,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.54500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.59999999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93400000000000005,"5min":0.997,"15min":0.997},"last":0.66900000000000004}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":411471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.312},"min":{"1min":0.25800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.311,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.29099999999999998}]},{"osd":1,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.52500000000000002,"15min":0.51600000000000001},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57599999999999996}]},{"osd":2,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.56799999999999995,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.52600000000000002},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.55600000000000005,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998}]},{"osd":5,"last update":"Fri Aug 25 02:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.69499999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Aug 25 02:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53300000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.35299999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:24:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.43099999999999999}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.628},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64600000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":411831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.309,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.38300000000000001,"15min":0.36099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.44700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.46600000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.442,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Aug 25 02:25:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.61499999999999999,"15min":0.58399999999999996},"min":{"1min":0.49299999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80800000000000005,"5min":1.097,"15min":1.097},"last":0.752},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56599999999999995}]},{"osd":8,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64200000000000002,"15min":0.63600000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97199999999999998,"5min":1.113,"15min":1.113},"last":0.60299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:25:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89100000000000001,"5min":1.131,"15min":1.131},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67500000000000004,"15min":0.66600000000000004},"min":{"1min":0.57799999999999996,"5min":0.373,"15min":0.373},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73999999999999999}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64400000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1002700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.745,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.73899999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.30499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.42299999999999999,"15min":0.41199999999999998},"min":{"1min":0.28399999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.622},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57999999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.8999999999999999,"5min":3.8999999999999999,"15min":3.8999999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.46300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81599999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.748},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.67600000000000005,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.72499999999999998}]},{"osd":8,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69599999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69599999999999995,"15min":0.68400000000000005},"min":{"1min":0.51100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72899999999999998,"15min":0.71699999999999997},"min":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76500000000000001}]},{"osd":10,"last update":"Fri Aug 25 02:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T02:25:47.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph pg dump --format=json 2023-08-25T02:25:47.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:47 smithi089 ceph-mon[98732]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:47.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:47 smithi121 ceph-mon[100099]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:47.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:47 smithi122 ceph-mon[100457]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:47.640 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:48.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:48 smithi121 ceph-mon[100099]: pgmap v272: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:48.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:48 smithi122 ceph-mon[100457]: pgmap v272: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:48.567 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:48 smithi089 ceph-mon[98732]: pgmap v272: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- 172.21.15.89:0/1087774166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 msgr2=0x7fa45810d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1087774166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa45810d760 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b0d0 tx=0x7fa45000b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- 172.21.15.89:0/1087774166 shutdown_connections 2023-08-25T02:25:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1087774166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa45810d760 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1087774166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4580688e0 0x7fa458068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1087774166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa458105390 0x7fa458105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- 172.21.15.89:0/1087774166 >> 172.21.15.89:0/1087774166 conn(0x7fa458076920 msgr2=0x7fa458076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- 172.21.15.89:0/1087774166 shutdown_connections 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- 172.21.15.89:0/1087774166 wait complete. 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 Processor -- start 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.748+0000 7fa45dd43700 1 -- start start 2023-08-25T02:25:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 0x7fa45819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa458105390 0x7fa45819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa458196f80 con 0x7fa458105d50 2023-08-25T02:25:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4581970f0 con 0x7fa4580688e0 2023-08-25T02:25:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4581972b0 con 0x7fa458105390 2023-08-25T02:25:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:48.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47220/0 (socket says 172.21.15.89:47220) 2023-08-25T02:25:48.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 -- 172.21.15.89:0/1379181926 learned_addr learned my addr 172.21.15.89:0/1379181926 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:48.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa4577fe700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 0x7fa45819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:48.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa456ffd700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa458105390 0x7fa45819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:48.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 -- 172.21.15.89:0/1379181926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa458105390 msgr2=0x7fa45819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:48.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa458105390 0x7fa45819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 -- 172.21.15.89:0/1379181926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 msgr2=0x7fa45819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:48.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 0x7fa45819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 -- 172.21.15.89:0/1379181926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450009d00 con 0x7fa458105d50 2023-08-25T02:25:48.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa4577fe700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 0x7fa45819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:48.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa457fff700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fd0 tx=0x7fa450007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:48.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45000e040 con 0x7fa458105d50 2023-08-25T02:25:48.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa458197530 con 0x7fa458105d50 2023-08-25T02:25:48.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa450004e40 con 0x7fa458105d50 2023-08-25T02:25:48.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.749+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450012a50 con 0x7fa458105d50 2023-08-25T02:25:48.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.750+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581a3880 con 0x7fa458105d50 2023-08-25T02:25:48.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.750+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45804e620 con 0x7fa458105d50 2023-08-25T02:25:48.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.751+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fa450005710 con 0x7fa458105d50 2023-08-25T02:25:48.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.752+0000 7fa454ff9700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 0x7fa4400a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:48.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.752+0000 7fa4577fe700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 0x7fa4400a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:48.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.752+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa4500cce20 con 0x7fa458105d50 2023-08-25T02:25:48.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.752+0000 7fa4577fe700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 0x7fa4400a2410 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0099d0 tx=0x7fa44c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:48.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.755+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa450096d90 con 0x7fa458105d50 2023-08-25T02:25:48.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.942+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa458110a00 con 0x7fa44009ff50 2023-08-25T02:25:48.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.951+0000 7fa454ff9700 1 -- 172.21.15.89:0/1379181926 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340707 (secure 0 0 0) 0x7fa458110a00 con 0x7fa44009ff50 2023-08-25T02:25:48.957 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:48.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 msgr2=0x7fa4400a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:48.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 0x7fa4400a2410 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0099d0 tx=0x7fa44c008040 comp rx=0 tx=0).stop 2023-08-25T02:25:48.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 msgr2=0x7fa458196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:48.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fd0 tx=0x7fa450007920 comp rx=0 tx=0).stop 2023-08-25T02:25:48.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 shutdown_connections 2023-08-25T02:25:48.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa44009ff50 0x7fa4400a2410 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4580688e0 0x7fa45819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.954+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa458105d50 0x7fa458196910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.955+0000 7fa45dd43700 1 --2- 172.21.15.89:0/1379181926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa458105390 0x7fa45819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.955+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 >> 172.21.15.89:0/1379181926 conn(0x7fa458076920 msgr2=0x7fa4580fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.955+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 shutdown_connections 2023-08-25T02:25:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:48.955+0000 7fa45dd43700 1 -- 172.21.15.89:0/1379181926 wait complete. 2023-08-25T02:25:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-08-25T02:25:49.399 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-08-25T02:25:47.308193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":371,"num_read_kb":250,"num_write":306,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336216,"kb_used_data":12004,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605608,"statfs":{"total":1151940427776,"available":1151596142592,"internally_reserved":0,"allocated":12292096,"data_stored":6720410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003032"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214182+0000","last_change":"2023-08-25T02:25:04.226503+0000","last_active":"2023-08-25T02:25:05.214182+0000","last_peered":"2023-08-25T02:25:05.214182+0000","last_clean":"2023-08-25T02:25:05.214182+0000","last_became_active":"2023-08-25T02:25:04.226184+0000","last_became_peered":"2023-08-25T02:25:04.226184+0000","last_unstale":"2023-08-25T02:25:05.214182+0000","last_undegraded":"2023-08-25T02:25:05.214182+0000","last_fullsized":"2023-08-25T02:25:05.214182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:17:33.775196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938361+0000","last_change":"2023-08-25T02:25:04.236479+0000","last_active":"2023-08-25T02:25:05.938361+0000","last_peered":"2023-08-25T02:25:05.938361+0000","last_clean":"2023-08-25T02:25:05.938361+0000","last_became_active":"2023-08-25T02:25:04.235376+0000","last_became_peered":"2023-08-25T02:25:04.235376+0000","last_unstale":"2023-08-25T02:25:05.938361+0000","last_undegraded":"2023-08-25T02:25:05.938361+0000","last_fullsized":"2023-08-25T02:25:05.938361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:47.418998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938362+0000","last_change":"2023-08-25T02:25:04.235840+0000","last_active":"2023-08-25T02:25:05.938362+0000","last_peered":"2023-08-25T02:25:05.938362+0000","last_clean":"2023-08-25T02:25:05.938362+0000","last_became_active":"2023-08-25T02:25:04.235279+0000","last_became_peered":"2023-08-25T02:25:04.235279+0000","last_unstale":"2023-08-25T02:25:05.938362+0000","last_undegraded":"2023-08-25T02:25:05.938362+0000","last_fullsized":"2023-08-25T02:25:05.938362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:23.103019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474317+0000","last_change":"2023-08-25T02:25:04.237793+0000","last_active":"2023-08-25T02:25:06.474317+0000","last_peered":"2023-08-25T02:25:06.474317+0000","last_clean":"2023-08-25T02:25:06.474317+0000","last_became_active":"2023-08-25T02:25:04.237493+0000","last_became_peered":"2023-08-25T02:25:04.237493+0000","last_unstale":"2023-08-25T02:25:06.474317+0000","last_undegraded":"2023-08-25T02:25:06.474317+0000","last_fullsized":"2023-08-25T02:25:06.474317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:36:34.299231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657466+0000","last_change":"2023-08-25T02:25:04.221053+0000","last_active":"2023-08-25T02:25:06.657466+0000","last_peered":"2023-08-25T02:25:06.657466+0000","last_clean":"2023-08-25T02:25:06.657466+0000","last_became_active":"2023-08-25T02:25:04.220710+0000","last_became_peered":"2023-08-25T02:25:04.220710+0000","last_unstale":"2023-08-25T02:25:06.657466+0000","last_undegraded":"2023-08-25T02:25:06.657466+0000","last_fullsized":"2023-08-25T02:25:06.657466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T02:25:14.107245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214409+0000","last_change":"2023-08-25T02:25:04.234380+0000","last_active":"2023-08-25T02:25:05.214409+0000","last_peered":"2023-08-25T02:25:05.214409+0000","last_clean":"2023-08-25T02:25:05.214409+0000","last_became_active":"2023-08-25T02:25:04.233376+0000","last_became_peered":"2023-08-25T02:25:04.233376+0000","last_unstale":"2023-08-25T02:25:05.214409+0000","last_undegraded":"2023-08-25T02:25:05.214409+0000","last_fullsized":"2023-08-25T02:25:05.214409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:18:49.891288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473517+0000","last_change":"2023-08-25T02:25:04.228398+0000","last_active":"2023-08-25T02:25:06.473517+0000","last_peered":"2023-08-25T02:25:06.473517+0000","last_clean":"2023-08-25T02:25:06.473517+0000","last_became_active":"2023-08-25T02:25:04.227310+0000","last_became_peered":"2023-08-25T02:25:04.227310+0000","last_unstale":"2023-08-25T02:25:06.473517+0000","last_undegraded":"2023-08-25T02:25:06.473517+0000","last_fullsized":"2023-08-25T02:25:06.473517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:45:46.502139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213776+0000","last_change":"2023-08-25T02:25:04.235175+0000","last_active":"2023-08-25T02:25:05.213776+0000","last_peered":"2023-08-25T02:25:05.213776+0000","last_clean":"2023-08-25T02:25:05.213776+0000","last_became_active":"2023-08-25T02:25:04.234048+0000","last_became_peered":"2023-08-25T02:25:04.234048+0000","last_unstale":"2023-08-25T02:25:05.213776+0000","last_undegraded":"2023-08-25T02:25:05.213776+0000","last_fullsized":"2023-08-25T02:25:05.213776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:20.347389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438643+0000","last_change":"2023-08-25T02:25:04.226475+0000","last_active":"2023-08-25T02:25:06.438643+0000","last_peered":"2023-08-25T02:25:06.438643+0000","last_clean":"2023-08-25T02:25:06.438643+0000","last_became_active":"2023-08-25T02:25:04.226125+0000","last_became_peered":"2023-08-25T02:25:04.226125+0000","last_unstale":"2023-08-25T02:25:06.438643+0000","last_undegraded":"2023-08-25T02:25:06.438643+0000","last_fullsized":"2023-08-25T02:25:06.438643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:44.888916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214553+0000","last_change":"2023-08-25T02:25:04.234476+0000","last_active":"2023-08-25T02:25:05.214553+0000","last_peered":"2023-08-25T02:25:05.214553+0000","last_clean":"2023-08-25T02:25:05.214553+0000","last_became_active":"2023-08-25T02:25:04.234062+0000","last_became_peered":"2023-08-25T02:25:04.234062+0000","last_unstale":"2023-08-25T02:25:05.214553+0000","last_undegraded":"2023-08-25T02:25:05.214553+0000","last_fullsized":"2023-08-25T02:25:05.214553+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:35:10.280467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473970+0000","last_change":"2023-08-25T02:25:04.233218+0000","last_active":"2023-08-25T02:25:06.473970+0000","last_peered":"2023-08-25T02:25:06.473970+0000","last_clean":"2023-08-25T02:25:06.473970+0000","last_became_active":"2023-08-25T02:25:04.232933+0000","last_became_peered":"2023-08-25T02:25:04.232933+0000","last_unstale":"2023-08-25T02:25:06.473970+0000","last_undegraded":"2023-08-25T02:25:06.473970+0000","last_fullsized":"2023-08-25T02:25:06.473970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T03:32:06.548589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474291+0000","last_change":"2023-08-25T02:25:04.238646+0000","last_active":"2023-08-25T02:25:06.474291+0000","last_peered":"2023-08-25T02:25:06.474291+0000","last_clean":"2023-08-25T02:25:06.474291+0000","last_became_active":"2023-08-25T02:25:04.238252+0000","last_became_peered":"2023-08-25T02:25:04.238252+0000","last_unstale":"2023-08-25T02:25:06.474291+0000","last_undegraded":"2023-08-25T02:25:06.474291+0000","last_fullsized":"2023-08-25T02:25:06.474291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:43:32.951632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473441+0000","last_change":"2023-08-25T02:25:04.236025+0000","last_active":"2023-08-25T02:25:06.473441+0000","last_peered":"2023-08-25T02:25:06.473441+0000","last_clean":"2023-08-25T02:25:06.473441+0000","last_became_active":"2023-08-25T02:25:04.235682+0000","last_became_peered":"2023-08-25T02:25:04.235682+0000","last_unstale":"2023-08-25T02:25:06.473441+0000","last_undegraded":"2023-08-25T02:25:06.473441+0000","last_fullsized":"2023-08-25T02:25:06.473441+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:29:46.263361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938102+0000","last_change":"2023-08-25T02:25:04.236295+0000","last_active":"2023-08-25T02:25:05.938102+0000","last_peered":"2023-08-25T02:25:05.938102+0000","last_clean":"2023-08-25T02:25:05.938102+0000","last_became_active":"2023-08-25T02:25:04.235979+0000","last_became_peered":"2023-08-25T02:25:04.235979+0000","last_unstale":"2023-08-25T02:25:05.938102+0000","last_undegraded":"2023-08-25T02:25:05.938102+0000","last_fullsized":"2023-08-25T02:25:05.938102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:12:13.224551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938095+0000","last_change":"2023-08-25T02:25:04.234885+0000","last_active":"2023-08-25T02:25:05.938095+0000","last_peered":"2023-08-25T02:25:05.938095+0000","last_clean":"2023-08-25T02:25:05.938095+0000","last_became_active":"2023-08-25T02:25:04.234603+0000","last_became_peered":"2023-08-25T02:25:04.234603+0000","last_unstale":"2023-08-25T02:25:05.938095+0000","last_undegraded":"2023-08-25T02:25:05.938095+0000","last_fullsized":"2023-08-25T02:25:05.938095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:26.095674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438191+0000","last_change":"2023-08-25T02:25:04.230151+0000","last_active":"2023-08-25T02:25:06.438191+0000","last_peered":"2023-08-25T02:25:06.438191+0000","last_clean":"2023-08-25T02:25:06.438191+0000","last_became_active":"2023-08-25T02:25:04.227881+0000","last_became_peered":"2023-08-25T02:25:04.227881+0000","last_unstale":"2023-08-25T02:25:06.438191+0000","last_undegraded":"2023-08-25T02:25:06.438191+0000","last_fullsized":"2023-08-25T02:25:06.438191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:04:31.382761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214311+0000","last_change":"2023-08-25T02:25:04.230117+0000","last_active":"2023-08-25T02:25:05.214311+0000","last_peered":"2023-08-25T02:25:05.214311+0000","last_clean":"2023-08-25T02:25:05.214311+0000","last_became_active":"2023-08-25T02:25:04.229810+0000","last_became_peered":"2023-08-25T02:25:04.229810+0000","last_unstale":"2023-08-25T02:25:05.214311+0000","last_undegraded":"2023-08-25T02:25:05.214311+0000","last_fullsized":"2023-08-25T02:25:05.214311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:54:12.194687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473816+0000","last_change":"2023-08-25T02:25:04.235011+0000","last_active":"2023-08-25T02:25:06.473816+0000","last_peered":"2023-08-25T02:25:06.473816+0000","last_clean":"2023-08-25T02:25:06.473816+0000","last_became_active":"2023-08-25T02:25:04.234664+0000","last_became_peered":"2023-08-25T02:25:04.234664+0000","last_unstale":"2023-08-25T02:25:06.473816+0000","last_undegraded":"2023-08-25T02:25:06.473816+0000","last_fullsized":"2023-08-25T02:25:06.473816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:50:31.186711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473535+0000","last_change":"2023-08-25T02:25:04.229223+0000","last_active":"2023-08-25T02:25:06.473535+0000","last_peered":"2023-08-25T02:25:06.473535+0000","last_clean":"2023-08-25T02:25:06.473535+0000","last_became_active":"2023-08-25T02:25:04.227656+0000","last_became_peered":"2023-08-25T02:25:04.227656+0000","last_unstale":"2023-08-25T02:25:06.473535+0000","last_undegraded":"2023-08-25T02:25:06.473535+0000","last_fullsized":"2023-08-25T02:25:06.473535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:50:59.352227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'9","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640903+0000","last_change":"2023-08-25T02:25:03.613357+0000","last_active":"2023-08-25T02:25:08.640903+0000","last_peered":"2023-08-25T02:25:08.640903+0000","last_clean":"2023-08-25T02:25:08.640903+0000","last_became_active":"2023-08-25T02:25:03.612977+0000","last_became_peered":"2023-08-25T02:25:03.612977+0000","last_unstale":"2023-08-25T02:25:08.640903+0000","last_undegraded":"2023-08-25T02:25:08.640903+0000","last_fullsized":"2023-08-25T02:25:08.640903+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:28:01.586298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214090+0000","last_change":"2023-08-25T02:25:04.230544+0000","last_active":"2023-08-25T02:25:05.214090+0000","last_peered":"2023-08-25T02:25:05.214090+0000","last_clean":"2023-08-25T02:25:05.214090+0000","last_became_active":"2023-08-25T02:25:04.230226+0000","last_became_peered":"2023-08-25T02:25:04.230226+0000","last_unstale":"2023-08-25T02:25:05.214090+0000","last_undegraded":"2023-08-25T02:25:05.214090+0000","last_fullsized":"2023-08-25T02:25:05.214090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:41:41.647707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.830079+0000","last_change":"2023-08-25T02:25:03.617626+0000","last_active":"2023-08-25T02:25:05.830079+0000","last_peered":"2023-08-25T02:25:05.830079+0000","last_clean":"2023-08-25T02:25:05.830079+0000","last_became_active":"2023-08-25T02:25:03.617323+0000","last_became_peered":"2023-08-25T02:25:03.617323+0000","last_unstale":"2023-08-25T02:25:05.830079+0000","last_undegraded":"2023-08-25T02:25:05.830079+0000","last_fullsized":"2023-08-25T02:25:05.830079+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:05.360379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938528+0000","last_change":"2023-08-25T02:25:04.226808+0000","last_active":"2023-08-25T02:25:05.938528+0000","last_peered":"2023-08-25T02:25:05.938528+0000","last_clean":"2023-08-25T02:25:05.938528+0000","last_became_active":"2023-08-25T02:25:04.226407+0000","last_became_peered":"2023-08-25T02:25:04.226407+0000","last_unstale":"2023-08-25T02:25:05.938528+0000","last_undegraded":"2023-08-25T02:25:05.938528+0000","last_fullsized":"2023-08-25T02:25:05.938528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:26:23.478853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.639252+0000","last_change":"2023-08-25T02:25:03.611831+0000","last_active":"2023-08-25T02:25:08.639252+0000","last_peered":"2023-08-25T02:25:08.639252+0000","last_clean":"2023-08-25T02:25:08.639252+0000","last_became_active":"2023-08-25T02:25:03.611530+0000","last_became_peered":"2023-08-25T02:25:03.611530+0000","last_unstale":"2023-08-25T02:25:08.639252+0000","last_undegraded":"2023-08-25T02:25:08.639252+0000","last_fullsized":"2023-08-25T02:25:08.639252+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:02:49.263476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438337+0000","last_change":"2023-08-25T02:25:04.236572+0000","last_active":"2023-08-25T02:25:06.438337+0000","last_peered":"2023-08-25T02:25:06.438337+0000","last_clean":"2023-08-25T02:25:06.438337+0000","last_became_active":"2023-08-25T02:25:04.236242+0000","last_became_peered":"2023-08-25T02:25:04.236242+0000","last_unstale":"2023-08-25T02:25:06.438337+0000","last_undegraded":"2023-08-25T02:25:06.438337+0000","last_fullsized":"2023-08-25T02:25:06.438337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:45:50.267973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657579+0000","last_change":"2023-08-25T02:25:03.616265+0000","last_active":"2023-08-25T02:25:06.657579+0000","last_peered":"2023-08-25T02:25:06.657579+0000","last_clean":"2023-08-25T02:25:06.657579+0000","last_became_active":"2023-08-25T02:25:03.615926+0000","last_became_peered":"2023-08-25T02:25:03.615926+0000","last_unstale":"2023-08-25T02:25:06.657579+0000","last_undegraded":"2023-08-25T02:25:06.657579+0000","last_fullsized":"2023-08-25T02:25:06.657579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:21:34.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157129+0000","last_change":"2023-08-25T02:25:04.232925+0000","last_active":"2023-08-25T02:25:06.157129+0000","last_peered":"2023-08-25T02:25:06.157129+0000","last_clean":"2023-08-25T02:25:06.157129+0000","last_became_active":"2023-08-25T02:25:04.232615+0000","last_became_peered":"2023-08-25T02:25:04.232615+0000","last_unstale":"2023-08-25T02:25:06.157129+0000","last_undegraded":"2023-08-25T02:25:06.157129+0000","last_fullsized":"2023-08-25T02:25:06.157129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:48:54.025387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:41.696058+0000","last_change":"2023-08-25T02:25:03.618534+0000","last_active":"2023-08-25T02:25:41.696058+0000","last_peered":"2023-08-25T02:25:41.696058+0000","last_clean":"2023-08-25T02:25:41.696058+0000","last_became_active":"2023-08-25T02:25:03.618284+0000","last_became_peered":"2023-08-25T02:25:03.618284+0000","last_unstale":"2023-08-25T02:25:41.696058+0000","last_undegraded":"2023-08-25T02:25:41.696058+0000","last_fullsized":"2023-08-25T02:25:41.696058+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:55:34.354817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657492+0000","last_change":"2023-08-25T02:25:04.225148+0000","last_active":"2023-08-25T02:25:06.657492+0000","last_peered":"2023-08-25T02:25:06.657492+0000","last_clean":"2023-08-25T02:25:06.657492+0000","last_became_active":"2023-08-25T02:25:04.224776+0000","last_became_peered":"2023-08-25T02:25:04.224776+0000","last_unstale":"2023-08-25T02:25:06.657492+0000","last_undegraded":"2023-08-25T02:25:06.657492+0000","last_fullsized":"2023-08-25T02:25:06.657492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:27.220482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641178+0000","last_change":"2023-08-25T02:25:03.610869+0000","last_active":"2023-08-25T02:25:08.641178+0000","last_peered":"2023-08-25T02:25:08.641178+0000","last_clean":"2023-08-25T02:25:08.641178+0000","last_became_active":"2023-08-25T02:25:03.610568+0000","last_became_peered":"2023-08-25T02:25:03.610568+0000","last_unstale":"2023-08-25T02:25:08.641178+0000","last_undegraded":"2023-08-25T02:25:08.641178+0000","last_fullsized":"2023-08-25T02:25:08.641178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:01:16.686802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157177+0000","last_change":"2023-08-25T02:25:04.230241+0000","last_active":"2023-08-25T02:25:06.157177+0000","last_peered":"2023-08-25T02:25:06.157177+0000","last_clean":"2023-08-25T02:25:06.157177+0000","last_became_active":"2023-08-25T02:25:04.229871+0000","last_became_peered":"2023-08-25T02:25:04.229871+0000","last_unstale":"2023-08-25T02:25:06.157177+0000","last_undegraded":"2023-08-25T02:25:06.157177+0000","last_fullsized":"2023-08-25T02:25:06.157177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:20:16.764817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641247+0000","last_change":"2023-08-25T02:25:03.610650+0000","last_active":"2023-08-25T02:25:08.641247+0000","last_peered":"2023-08-25T02:25:08.641247+0000","last_clean":"2023-08-25T02:25:08.641247+0000","last_became_active":"2023-08-25T02:25:03.609901+0000","last_became_peered":"2023-08-25T02:25:03.609901+0000","last_unstale":"2023-08-25T02:25:08.641247+0000","last_undegraded":"2023-08-25T02:25:08.641247+0000","last_fullsized":"2023-08-25T02:25:08.641247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:16.408779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213759+0000","last_change":"2023-08-25T02:25:04.222682+0000","last_active":"2023-08-25T02:25:05.213759+0000","last_peered":"2023-08-25T02:25:05.213759+0000","last_clean":"2023-08-25T02:25:05.213759+0000","last_became_active":"2023-08-25T02:25:04.221791+0000","last_became_peered":"2023-08-25T02:25:04.221791+0000","last_unstale":"2023-08-25T02:25:05.213759+0000","last_undegraded":"2023-08-25T02:25:05.213759+0000","last_fullsized":"2023-08-25T02:25:05.213759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:29:20.943592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156809+0000","last_change":"2023-08-25T02:25:03.610342+0000","last_active":"2023-08-25T02:25:06.156809+0000","last_peered":"2023-08-25T02:25:06.156809+0000","last_clean":"2023-08-25T02:25:06.156809+0000","last_became_active":"2023-08-25T02:25:03.609928+0000","last_became_peered":"2023-08-25T02:25:03.609928+0000","last_unstale":"2023-08-25T02:25:06.156809+0000","last_undegraded":"2023-08-25T02:25:06.156809+0000","last_fullsized":"2023-08-25T02:25:06.156809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:08:42.862108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473531+0000","last_change":"2023-08-25T02:25:04.234364+0000","last_active":"2023-08-25T02:25:06.473531+0000","last_peered":"2023-08-25T02:25:06.473531+0000","last_clean":"2023-08-25T02:25:06.473531+0000","last_became_active":"2023-08-25T02:25:04.234050+0000","last_became_peered":"2023-08-25T02:25:04.234050+0000","last_unstale":"2023-08-25T02:25:06.473531+0000","last_undegraded":"2023-08-25T02:25:06.473531+0000","last_fullsized":"2023-08-25T02:25:06.473531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:39:50.455964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:44.699581+0000","last_change":"2023-08-25T02:25:03.617016+0000","last_active":"2023-08-25T02:25:44.699581+0000","last_peered":"2023-08-25T02:25:44.699581+0000","last_clean":"2023-08-25T02:25:44.699581+0000","last_became_active":"2023-08-25T02:25:03.616670+0000","last_became_peered":"2023-08-25T02:25:03.616670+0000","last_unstale":"2023-08-25T02:25:44.699581+0000","last_undegraded":"2023-08-25T02:25:44.699581+0000","last_fullsized":"2023-08-25T02:25:44.699581+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:42:03.302702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438625+0000","last_change":"2023-08-25T02:25:04.236009+0000","last_active":"2023-08-25T02:25:06.438625+0000","last_peered":"2023-08-25T02:25:06.438625+0000","last_clean":"2023-08-25T02:25:06.438625+0000","last_became_active":"2023-08-25T02:25:04.235722+0000","last_became_peered":"2023-08-25T02:25:04.235722+0000","last_unstale":"2023-08-25T02:25:06.438625+0000","last_undegraded":"2023-08-25T02:25:06.438625+0000","last_fullsized":"2023-08-25T02:25:06.438625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:42:44.656781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633826+0000","last_change":"2023-08-25T02:25:03.611060+0000","last_active":"2023-08-25T02:25:07.633826+0000","last_peered":"2023-08-25T02:25:07.633826+0000","last_clean":"2023-08-25T02:25:07.633826+0000","last_became_active":"2023-08-25T02:25:03.610715+0000","last_became_peered":"2023-08-25T02:25:03.610715+0000","last_unstale":"2023-08-25T02:25:07.633826+0000","last_undegraded":"2023-08-25T02:25:07.633826+0000","last_fullsized":"2023-08-25T02:25:07.633826+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:25:17.684716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938519+0000","last_change":"2023-08-25T02:25:04.225855+0000","last_active":"2023-08-25T02:25:05.938519+0000","last_peered":"2023-08-25T02:25:05.938519+0000","last_clean":"2023-08-25T02:25:05.938519+0000","last_became_active":"2023-08-25T02:25:04.225559+0000","last_became_peered":"2023-08-25T02:25:04.225559+0000","last_unstale":"2023-08-25T02:25:05.938519+0000","last_undegraded":"2023-08-25T02:25:05.938519+0000","last_fullsized":"2023-08-25T02:25:05.938519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:33:19.054268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632104+0000","last_change":"2023-08-25T02:25:03.617097+0000","last_active":"2023-08-25T02:25:07.632104+0000","last_peered":"2023-08-25T02:25:07.632104+0000","last_clean":"2023-08-25T02:25:07.632104+0000","last_became_active":"2023-08-25T02:25:03.616809+0000","last_became_peered":"2023-08-25T02:25:03.616809+0000","last_unstale":"2023-08-25T02:25:07.632104+0000","last_undegraded":"2023-08-25T02:25:07.632104+0000","last_fullsized":"2023-08-25T02:25:07.632104+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:11:24.908899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438483+0000","last_change":"2023-08-25T02:25:04.226384+0000","last_active":"2023-08-25T02:25:06.438483+0000","last_peered":"2023-08-25T02:25:06.438483+0000","last_clean":"2023-08-25T02:25:06.438483+0000","last_became_active":"2023-08-25T02:25:04.226047+0000","last_became_peered":"2023-08-25T02:25:04.226047+0000","last_unstale":"2023-08-25T02:25:06.438483+0000","last_undegraded":"2023-08-25T02:25:06.438483+0000","last_fullsized":"2023-08-25T02:25:06.438483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:45:01.485924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'8","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:45.701085+0000","last_change":"2023-08-25T02:25:03.609570+0000","last_active":"2023-08-25T02:25:45.701085+0000","last_peered":"2023-08-25T02:25:45.701085+0000","last_clean":"2023-08-25T02:25:45.701085+0000","last_became_active":"2023-08-25T02:25:03.609253+0000","last_became_peered":"2023-08-25T02:25:03.609253+0000","last_unstale":"2023-08-25T02:25:45.701085+0000","last_undegraded":"2023-08-25T02:25:45.701085+0000","last_fullsized":"2023-08-25T02:25:45.701085+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:14.775308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":39,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657588+0000","last_change":"2023-08-25T02:25:04.226667+0000","last_active":"2023-08-25T02:25:06.657588+0000","last_peered":"2023-08-25T02:25:06.657588+0000","last_clean":"2023-08-25T02:25:06.657588+0000","last_became_active":"2023-08-25T02:25:04.226332+0000","last_became_peered":"2023-08-25T02:25:04.226332+0000","last_unstale":"2023-08-25T02:25:06.657588+0000","last_undegraded":"2023-08-25T02:25:06.657588+0000","last_fullsized":"2023-08-25T02:25:06.657588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:51:57.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640547+0000","last_change":"2023-08-25T02:25:03.609419+0000","last_active":"2023-08-25T02:25:08.640547+0000","last_peered":"2023-08-25T02:25:08.640547+0000","last_clean":"2023-08-25T02:25:08.640547+0000","last_became_active":"2023-08-25T02:25:03.609101+0000","last_became_peered":"2023-08-25T02:25:03.609101+0000","last_unstale":"2023-08-25T02:25:08.640547+0000","last_undegraded":"2023-08-25T02:25:08.640547+0000","last_fullsized":"2023-08-25T02:25:08.640547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:15.692390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473753+0000","last_change":"2023-08-25T02:25:04.231351+0000","last_active":"2023-08-25T02:25:06.473753+0000","last_peered":"2023-08-25T02:25:06.473753+0000","last_clean":"2023-08-25T02:25:06.473753+0000","last_became_active":"2023-08-25T02:25:04.230974+0000","last_became_peered":"2023-08-25T02:25:04.230974+0000","last_unstale":"2023-08-25T02:25:06.473753+0000","last_undegraded":"2023-08-25T02:25:06.473753+0000","last_fullsized":"2023-08-25T02:25:06.473753+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:12.882322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633217+0000","last_change":"2023-08-25T02:25:03.617889+0000","last_active":"2023-08-25T02:25:07.633217+0000","last_peered":"2023-08-25T02:25:07.633217+0000","last_clean":"2023-08-25T02:25:07.633217+0000","last_became_active":"2023-08-25T02:25:03.617576+0000","last_became_peered":"2023-08-25T02:25:03.617576+0000","last_unstale":"2023-08-25T02:25:07.633217+0000","last_undegraded":"2023-08-25T02:25:07.633217+0000","last_fullsized":"2023-08-25T02:25:07.633217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:38:22.627438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473924+0000","last_change":"2023-08-25T02:25:04.230135+0000","last_active":"2023-08-25T02:25:06.473924+0000","last_peered":"2023-08-25T02:25:06.473924+0000","last_clean":"2023-08-25T02:25:06.473924+0000","last_became_active":"2023-08-25T02:25:04.229592+0000","last_became_peered":"2023-08-25T02:25:04.229592+0000","last_unstale":"2023-08-25T02:25:06.473924+0000","last_undegraded":"2023-08-25T02:25:06.473924+0000","last_fullsized":"2023-08-25T02:25:06.473924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:28:47.602421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'3","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:40.693631+0000","last_change":"2023-08-25T02:25:03.610021+0000","last_active":"2023-08-25T02:25:40.693631+0000","last_peered":"2023-08-25T02:25:40.693631+0000","last_clean":"2023-08-25T02:25:40.693631+0000","last_became_active":"2023-08-25T02:25:03.609703+0000","last_became_peered":"2023-08-25T02:25:03.609703+0000","last_unstale":"2023-08-25T02:25:40.693631+0000","last_undegraded":"2023-08-25T02:25:40.693631+0000","last_fullsized":"2023-08-25T02:25:40.693631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:25.169266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473493+0000","last_change":"2023-08-25T02:25:04.234643+0000","last_active":"2023-08-25T02:25:06.473493+0000","last_peered":"2023-08-25T02:25:06.473493+0000","last_clean":"2023-08-25T02:25:06.473493+0000","last_became_active":"2023-08-25T02:25:04.234349+0000","last_became_peered":"2023-08-25T02:25:04.234349+0000","last_unstale":"2023-08-25T02:25:06.473493+0000","last_undegraded":"2023-08-25T02:25:06.473493+0000","last_fullsized":"2023-08-25T02:25:06.473493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:08:25.591726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.640938+0000","last_change":"2023-08-25T02:25:03.613702+0000","last_active":"2023-08-25T02:25:08.640938+0000","last_peered":"2023-08-25T02:25:08.640938+0000","last_clean":"2023-08-25T02:25:08.640938+0000","last_became_active":"2023-08-25T02:25:03.613383+0000","last_became_peered":"2023-08-25T02:25:03.613383+0000","last_unstale":"2023-08-25T02:25:08.640938+0000","last_undegraded":"2023-08-25T02:25:08.640938+0000","last_fullsized":"2023-08-25T02:25:08.640938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:53.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156815+0000","last_change":"2023-08-25T02:25:04.227650+0000","last_active":"2023-08-25T02:25:06.156815+0000","last_peered":"2023-08-25T02:25:06.156815+0000","last_clean":"2023-08-25T02:25:06.156815+0000","last_became_active":"2023-08-25T02:25:04.227335+0000","last_became_peered":"2023-08-25T02:25:04.227335+0000","last_unstale":"2023-08-25T02:25:06.156815+0000","last_undegraded":"2023-08-25T02:25:06.156815+0000","last_fullsized":"2023-08-25T02:25:06.156815+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:10.086931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:09.643003+0000","last_change":"2023-08-25T02:25:03.616065+0000","last_active":"2023-08-25T02:25:09.643003+0000","last_peered":"2023-08-25T02:25:09.643003+0000","last_clean":"2023-08-25T02:25:09.643003+0000","last_became_active":"2023-08-25T02:25:03.615751+0000","last_became_peered":"2023-08-25T02:25:03.615751+0000","last_unstale":"2023-08-25T02:25:09.643003+0000","last_undegraded":"2023-08-25T02:25:09.643003+0000","last_fullsized":"2023-08-25T02:25:09.643003+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:27:27.917233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214184+0000","last_change":"2023-08-25T02:25:04.228285+0000","last_active":"2023-08-25T02:25:05.214184+0000","last_peered":"2023-08-25T02:25:05.214184+0000","last_clean":"2023-08-25T02:25:05.214184+0000","last_became_active":"2023-08-25T02:25:04.228061+0000","last_became_peered":"2023-08-25T02:25:04.228061+0000","last_unstale":"2023-08-25T02:25:05.214184+0000","last_undegraded":"2023-08-25T02:25:05.214184+0000","last_fullsized":"2023-08-25T02:25:05.214184+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:40.994665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473497+0000","last_change":"2023-08-25T02:25:03.615712+0000","last_active":"2023-08-25T02:25:06.473497+0000","last_peered":"2023-08-25T02:25:06.473497+0000","last_clean":"2023-08-25T02:25:06.473497+0000","last_became_active":"2023-08-25T02:25:03.615412+0000","last_became_peered":"2023-08-25T02:25:03.615412+0000","last_unstale":"2023-08-25T02:25:06.473497+0000","last_undegraded":"2023-08-25T02:25:06.473497+0000","last_fullsized":"2023-08-25T02:25:06.473497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:14:19.940404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214408+0000","last_change":"2023-08-25T02:25:04.233485+0000","last_active":"2023-08-25T02:25:05.214408+0000","last_peered":"2023-08-25T02:25:05.214408+0000","last_clean":"2023-08-25T02:25:05.214408+0000","last_became_active":"2023-08-25T02:25:04.233205+0000","last_became_peered":"2023-08-25T02:25:04.233205+0000","last_unstale":"2023-08-25T02:25:05.214408+0000","last_undegraded":"2023-08-25T02:25:05.214408+0000","last_fullsized":"2023-08-25T02:25:05.214408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:12.219109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.413812+0000","last_change":"2023-08-25T02:25:03.613848+0000","last_active":"2023-08-25T02:25:07.413812+0000","last_peered":"2023-08-25T02:25:07.413812+0000","last_clean":"2023-08-25T02:25:07.413812+0000","last_became_active":"2023-08-25T02:25:03.613538+0000","last_became_peered":"2023-08-25T02:25:03.613538+0000","last_unstale":"2023-08-25T02:25:07.413812+0000","last_undegraded":"2023-08-25T02:25:07.413812+0000","last_fullsized":"2023-08-25T02:25:07.413812+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:52.883965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214475+0000","last_change":"2023-08-25T02:25:04.234167+0000","last_active":"2023-08-25T02:25:05.214475+0000","last_peered":"2023-08-25T02:25:05.214475+0000","last_clean":"2023-08-25T02:25:05.214475+0000","last_became_active":"2023-08-25T02:25:04.233779+0000","last_became_peered":"2023-08-25T02:25:04.233779+0000","last_unstale":"2023-08-25T02:25:05.214475+0000","last_undegraded":"2023-08-25T02:25:05.214475+0000","last_fullsized":"2023-08-25T02:25:05.214475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:48:19.007690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156909+0000","last_change":"2023-08-25T02:25:04.227759+0000","last_active":"2023-08-25T02:25:06.156909+0000","last_peered":"2023-08-25T02:25:06.156909+0000","last_clean":"2023-08-25T02:25:06.156909+0000","last_became_active":"2023-08-25T02:25:04.227166+0000","last_became_peered":"2023-08-25T02:25:04.227166+0000","last_unstale":"2023-08-25T02:25:06.156909+0000","last_undegraded":"2023-08-25T02:25:06.156909+0000","last_fullsized":"2023-08-25T02:25:06.156909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:58.916927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:44.698798+0000","last_change":"2023-08-25T02:25:03.613384+0000","last_active":"2023-08-25T02:25:44.698798+0000","last_peered":"2023-08-25T02:25:44.698798+0000","last_clean":"2023-08-25T02:25:44.698798+0000","last_became_active":"2023-08-25T02:25:03.613097+0000","last_became_peered":"2023-08-25T02:25:03.613097+0000","last_unstale":"2023-08-25T02:25:44.698798+0000","last_undegraded":"2023-08-25T02:25:44.698798+0000","last_fullsized":"2023-08-25T02:25:44.698798+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:55:46.368591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214227+0000","last_change":"2023-08-25T02:25:04.225009+0000","last_active":"2023-08-25T02:25:05.214227+0000","last_peered":"2023-08-25T02:25:05.214227+0000","last_clean":"2023-08-25T02:25:05.214227+0000","last_became_active":"2023-08-25T02:25:04.224678+0000","last_became_peered":"2023-08-25T02:25:04.224678+0000","last_unstale":"2023-08-25T02:25:05.214227+0000","last_undegraded":"2023-08-25T02:25:05.214227+0000","last_fullsized":"2023-08-25T02:25:05.214227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:45:44.498691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'3","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:09.642186+0000","last_change":"2023-08-25T02:25:03.607620+0000","last_active":"2023-08-25T02:25:09.642186+0000","last_peered":"2023-08-25T02:25:09.642186+0000","last_clean":"2023-08-25T02:25:09.642186+0000","last_became_active":"2023-08-25T02:25:03.607329+0000","last_became_peered":"2023-08-25T02:25:03.607329+0000","last_unstale":"2023-08-25T02:25:09.642186+0000","last_undegraded":"2023-08-25T02:25:09.642186+0000","last_fullsized":"2023-08-25T02:25:09.642186+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:11:21.994627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473471+0000","last_change":"2023-08-25T02:24:05.131624+0000","last_active":"2023-08-25T02:25:06.473471+0000","last_peered":"2023-08-25T02:25:06.473471+0000","last_clean":"2023-08-25T02:25:06.473471+0000","last_became_active":"2023-08-25T02:24:05.131084+0000","last_became_peered":"2023-08-25T02:24:05.131084+0000","last_unstale":"2023-08-25T02:25:06.473471+0000","last_undegraded":"2023-08-25T02:25:06.473471+0000","last_fullsized":"2023-08-25T02:25:06.473471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:21:10.049896+0000","last_clean_scrub_stamp":"2023-08-25T02:21:10.049896+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:55:44.618723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473109+0000","last_change":"2023-08-25T02:25:04.231797+0000","last_active":"2023-08-25T02:25:06.473109+0000","last_peered":"2023-08-25T02:25:06.473109+0000","last_clean":"2023-08-25T02:25:06.473109+0000","last_became_active":"2023-08-25T02:25:04.231520+0000","last_became_peered":"2023-08-25T02:25:04.231520+0000","last_unstale":"2023-08-25T02:25:06.473109+0000","last_undegraded":"2023-08-25T02:25:06.473109+0000","last_fullsized":"2023-08-25T02:25:06.473109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:30:43.262427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.639630+0000","last_change":"2023-08-25T02:25:03.611517+0000","last_active":"2023-08-25T02:25:08.639630+0000","last_peered":"2023-08-25T02:25:08.639630+0000","last_clean":"2023-08-25T02:25:08.639630+0000","last_became_active":"2023-08-25T02:25:03.611205+0000","last_became_peered":"2023-08-25T02:25:03.611205+0000","last_unstale":"2023-08-25T02:25:08.639630+0000","last_undegraded":"2023-08-25T02:25:08.639630+0000","last_fullsized":"2023-08-25T02:25:08.639630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:04:13.071834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.631473+0000","last_change":"2023-08-25T02:25:03.613795+0000","last_active":"2023-08-25T02:25:07.631473+0000","last_peered":"2023-08-25T02:25:07.631473+0000","last_clean":"2023-08-25T02:25:07.631473+0000","last_became_active":"2023-08-25T02:25:03.613410+0000","last_became_peered":"2023-08-25T02:25:03.613410+0000","last_unstale":"2023-08-25T02:25:07.631473+0000","last_undegraded":"2023-08-25T02:25:07.631473+0000","last_fullsized":"2023-08-25T02:25:07.631473+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:47:37.646320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473432+0000","last_change":"2023-08-25T02:25:04.232186+0000","last_active":"2023-08-25T02:25:06.473432+0000","last_peered":"2023-08-25T02:25:06.473432+0000","last_clean":"2023-08-25T02:25:06.473432+0000","last_became_active":"2023-08-25T02:25:04.231853+0000","last_became_peered":"2023-08-25T02:25:04.231853+0000","last_unstale":"2023-08-25T02:25:06.473432+0000","last_undegraded":"2023-08-25T02:25:06.473432+0000","last_fullsized":"2023-08-25T02:25:06.473432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:07:47.816234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641720+0000","last_change":"2023-08-25T02:25:03.615541+0000","last_active":"2023-08-25T02:25:08.641720+0000","last_peered":"2023-08-25T02:25:08.641720+0000","last_clean":"2023-08-25T02:25:08.641720+0000","last_became_active":"2023-08-25T02:25:03.615196+0000","last_became_peered":"2023-08-25T02:25:03.615196+0000","last_unstale":"2023-08-25T02:25:08.641720+0000","last_undegraded":"2023-08-25T02:25:08.641720+0000","last_fullsized":"2023-08-25T02:25:08.641720+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:46:14.320885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214288+0000","last_change":"2023-08-25T02:25:04.227688+0000","last_active":"2023-08-25T02:25:05.214288+0000","last_peered":"2023-08-25T02:25:05.214288+0000","last_clean":"2023-08-25T02:25:05.214288+0000","last_became_active":"2023-08-25T02:25:04.227394+0000","last_became_peered":"2023-08-25T02:25:04.227394+0000","last_unstale":"2023-08-25T02:25:05.214288+0000","last_undegraded":"2023-08-25T02:25:05.214288+0000","last_fullsized":"2023-08-25T02:25:05.214288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:53:57.237749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633564+0000","last_change":"2023-08-25T02:25:03.608500+0000","last_active":"2023-08-25T02:25:07.633564+0000","last_peered":"2023-08-25T02:25:07.633564+0000","last_clean":"2023-08-25T02:25:07.633564+0000","last_became_active":"2023-08-25T02:25:03.608172+0000","last_became_peered":"2023-08-25T02:25:03.608172+0000","last_unstale":"2023-08-25T02:25:07.633564+0000","last_undegraded":"2023-08-25T02:25:07.633564+0000","last_fullsized":"2023-08-25T02:25:07.633564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:18:45.063308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938372+0000","last_change":"2023-08-25T02:25:04.226768+0000","last_active":"2023-08-25T02:25:05.938372+0000","last_peered":"2023-08-25T02:25:05.938372+0000","last_clean":"2023-08-25T02:25:05.938372+0000","last_became_active":"2023-08-25T02:25:04.226459+0000","last_became_peered":"2023-08-25T02:25:04.226459+0000","last_unstale":"2023-08-25T02:25:05.938372+0000","last_undegraded":"2023-08-25T02:25:05.938372+0000","last_fullsized":"2023-08-25T02:25:05.938372+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:04:52.374035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633766+0000","last_change":"2023-08-25T02:25:03.612306+0000","last_active":"2023-08-25T02:25:07.633766+0000","last_peered":"2023-08-25T02:25:07.633766+0000","last_clean":"2023-08-25T02:25:07.633766+0000","last_became_active":"2023-08-25T02:25:03.611922+0000","last_became_peered":"2023-08-25T02:25:03.611922+0000","last_unstale":"2023-08-25T02:25:07.633766+0000","last_undegraded":"2023-08-25T02:25:07.633766+0000","last_fullsized":"2023-08-25T02:25:07.633766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:14:09.728422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.657329+0000","last_change":"2023-08-25T02:25:04.235032+0000","last_active":"2023-08-25T02:25:06.657329+0000","last_peered":"2023-08-25T02:25:06.657329+0000","last_clean":"2023-08-25T02:25:06.657329+0000","last_became_active":"2023-08-25T02:25:04.234626+0000","last_became_peered":"2023-08-25T02:25:04.234626+0000","last_unstale":"2023-08-25T02:25:06.657329+0000","last_undegraded":"2023-08-25T02:25:06.657329+0000","last_fullsized":"2023-08-25T02:25:06.657329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:00.849934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.632365+0000","last_change":"2023-08-25T02:25:03.608869+0000","last_active":"2023-08-25T02:25:07.632365+0000","last_peered":"2023-08-25T02:25:07.632365+0000","last_clean":"2023-08-25T02:25:07.632365+0000","last_became_active":"2023-08-25T02:25:03.608530+0000","last_became_peered":"2023-08-25T02:25:03.608530+0000","last_unstale":"2023-08-25T02:25:07.632365+0000","last_undegraded":"2023-08-25T02:25:07.632365+0000","last_fullsized":"2023-08-25T02:25:07.632365+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:51:39.415744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213929+0000","last_change":"2023-08-25T02:25:04.231830+0000","last_active":"2023-08-25T02:25:05.213929+0000","last_peered":"2023-08-25T02:25:05.213929+0000","last_clean":"2023-08-25T02:25:05.213929+0000","last_became_active":"2023-08-25T02:25:04.231530+0000","last_became_peered":"2023-08-25T02:25:04.231530+0000","last_unstale":"2023-08-25T02:25:05.213929+0000","last_undegraded":"2023-08-25T02:25:05.213929+0000","last_fullsized":"2023-08-25T02:25:05.213929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:34:31.230104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473570+0000","last_change":"2023-08-25T02:25:03.610625+0000","last_active":"2023-08-25T02:25:06.473570+0000","last_peered":"2023-08-25T02:25:06.473570+0000","last_clean":"2023-08-25T02:25:06.473570+0000","last_became_active":"2023-08-25T02:25:03.610305+0000","last_became_peered":"2023-08-25T02:25:03.610305+0000","last_unstale":"2023-08-25T02:25:06.473570+0000","last_undegraded":"2023-08-25T02:25:06.473570+0000","last_fullsized":"2023-08-25T02:25:06.473570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:28:44.209156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214041+0000","last_change":"2023-08-25T02:25:04.234533+0000","last_active":"2023-08-25T02:25:05.214041+0000","last_peered":"2023-08-25T02:25:05.214041+0000","last_clean":"2023-08-25T02:25:05.214041+0000","last_became_active":"2023-08-25T02:25:04.234167+0000","last_became_peered":"2023-08-25T02:25:04.234167+0000","last_unstale":"2023-08-25T02:25:05.214041+0000","last_undegraded":"2023-08-25T02:25:05.214041+0000","last_fullsized":"2023-08-25T02:25:05.214041+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:23:55.422360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:07.633449+0000","last_change":"2023-08-25T02:25:03.610139+0000","last_active":"2023-08-25T02:25:07.633449+0000","last_peered":"2023-08-25T02:25:07.633449+0000","last_clean":"2023-08-25T02:25:07.633449+0000","last_became_active":"2023-08-25T02:25:03.609811+0000","last_became_peered":"2023-08-25T02:25:03.609811+0000","last_unstale":"2023-08-25T02:25:07.633449+0000","last_undegraded":"2023-08-25T02:25:07.633449+0000","last_fullsized":"2023-08-25T02:25:07.633449+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:01.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474201+0000","last_change":"2023-08-25T02:25:04.233179+0000","last_active":"2023-08-25T02:25:06.474201+0000","last_peered":"2023-08-25T02:25:06.474201+0000","last_clean":"2023-08-25T02:25:06.474201+0000","last_became_active":"2023-08-25T02:25:04.232874+0000","last_became_peered":"2023-08-25T02:25:04.232874+0000","last_unstale":"2023-08-25T02:25:06.474201+0000","last_undegraded":"2023-08-25T02:25:06.474201+0000","last_fullsized":"2023-08-25T02:25:06.474201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:59:54.317603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:08.641063+0000","last_change":"2023-08-25T02:25:03.619258+0000","last_active":"2023-08-25T02:25:08.641063+0000","last_peered":"2023-08-25T02:25:08.641063+0000","last_clean":"2023-08-25T02:25:08.641063+0000","last_became_active":"2023-08-25T02:25:03.618954+0000","last_became_peered":"2023-08-25T02:25:03.618954+0000","last_unstale":"2023-08-25T02:25:08.641063+0000","last_undegraded":"2023-08-25T02:25:08.641063+0000","last_fullsized":"2023-08-25T02:25:08.641063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:39:48.139135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474059+0000","last_change":"2023-08-25T02:25:04.232620+0000","last_active":"2023-08-25T02:25:06.474059+0000","last_peered":"2023-08-25T02:25:06.474059+0000","last_clean":"2023-08-25T02:25:06.474059+0000","last_became_active":"2023-08-25T02:25:04.232314+0000","last_became_peered":"2023-08-25T02:25:04.232314+0000","last_unstale":"2023-08-25T02:25:06.474059+0000","last_undegraded":"2023-08-25T02:25:06.474059+0000","last_fullsized":"2023-08-25T02:25:06.474059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:39:46.683556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.157121+0000","last_change":"2023-08-25T02:25:03.617794+0000","last_active":"2023-08-25T02:25:06.157121+0000","last_peered":"2023-08-25T02:25:06.157121+0000","last_clean":"2023-08-25T02:25:06.157121+0000","last_became_active":"2023-08-25T02:25:03.617521+0000","last_became_peered":"2023-08-25T02:25:03.617521+0000","last_unstale":"2023-08-25T02:25:06.157121+0000","last_undegraded":"2023-08-25T02:25:06.157121+0000","last_fullsized":"2023-08-25T02:25:06.157121+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:35:22.049419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474136+0000","last_change":"2023-08-25T02:25:04.229796+0000","last_active":"2023-08-25T02:25:06.474136+0000","last_peered":"2023-08-25T02:25:06.474136+0000","last_clean":"2023-08-25T02:25:06.474136+0000","last_became_active":"2023-08-25T02:25:04.227073+0000","last_became_peered":"2023-08-25T02:25:04.227073+0000","last_unstale":"2023-08-25T02:25:06.474136+0000","last_undegraded":"2023-08-25T02:25:06.474136+0000","last_fullsized":"2023-08-25T02:25:06.474136+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:06:43.826591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473306+0000","last_change":"2023-08-25T02:25:03.612702+0000","last_active":"2023-08-25T02:25:06.473306+0000","last_peered":"2023-08-25T02:25:06.473306+0000","last_clean":"2023-08-25T02:25:06.473306+0000","last_became_active":"2023-08-25T02:25:03.612482+0000","last_became_peered":"2023-08-25T02:25:03.612482+0000","last_unstale":"2023-08-25T02:25:06.473306+0000","last_undegraded":"2023-08-25T02:25:06.473306+0000","last_fullsized":"2023-08-25T02:25:06.473306+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:02.591490+0000","last_clean_scrub_stamp":"2023-08-25T02:25:02.591490+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:02:34.423511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.474025+0000","last_change":"2023-08-25T02:25:04.229892+0000","last_active":"2023-08-25T02:25:06.474025+0000","last_peered":"2023-08-25T02:25:06.474025+0000","last_clean":"2023-08-25T02:25:06.474025+0000","last_became_active":"2023-08-25T02:25:04.229581+0000","last_became_peered":"2023-08-25T02:25:04.229581+0000","last_unstale":"2023-08-25T02:25:06.474025+0000","last_undegraded":"2023-08-25T02:25:06.474025+0000","last_fullsized":"2023-08-25T02:25:06.474025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:15:54.252164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214023+0000","last_change":"2023-08-25T02:25:04.231992+0000","last_active":"2023-08-25T02:25:05.214023+0000","last_peered":"2023-08-25T02:25:05.214023+0000","last_clean":"2023-08-25T02:25:05.214023+0000","last_became_active":"2023-08-25T02:25:04.231676+0000","last_became_peered":"2023-08-25T02:25:04.231676+0000","last_unstale":"2023-08-25T02:25:05.214023+0000","last_undegraded":"2023-08-25T02:25:05.214023+0000","last_fullsized":"2023-08-25T02:25:05.214023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T06:47:35.254078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473483+0000","last_change":"2023-08-25T02:25:04.227123+0000","last_active":"2023-08-25T02:25:06.473483+0000","last_peered":"2023-08-25T02:25:06.473483+0000","last_clean":"2023-08-25T02:25:06.473483+0000","last_became_active":"2023-08-25T02:25:04.226828+0000","last_became_peered":"2023-08-25T02:25:04.226828+0000","last_unstale":"2023-08-25T02:25:06.473483+0000","last_undegraded":"2023-08-25T02:25:06.473483+0000","last_fullsized":"2023-08-25T02:25:06.473483+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:22.044584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473472+0000","last_change":"2023-08-25T02:25:04.232635+0000","last_active":"2023-08-25T02:25:06.473472+0000","last_peered":"2023-08-25T02:25:06.473472+0000","last_clean":"2023-08-25T02:25:06.473472+0000","last_became_active":"2023-08-25T02:25:04.232353+0000","last_became_peered":"2023-08-25T02:25:04.232353+0000","last_unstale":"2023-08-25T02:25:06.473472+0000","last_undegraded":"2023-08-25T02:25:06.473472+0000","last_fullsized":"2023-08-25T02:25:06.473472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:14:24.041499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.938254+0000","last_change":"2023-08-25T02:25:04.226977+0000","last_active":"2023-08-25T02:25:05.938254+0000","last_peered":"2023-08-25T02:25:05.938254+0000","last_clean":"2023-08-25T02:25:05.938254+0000","last_became_active":"2023-08-25T02:25:04.226579+0000","last_became_peered":"2023-08-25T02:25:04.226579+0000","last_unstale":"2023-08-25T02:25:05.938254+0000","last_undegraded":"2023-08-25T02:25:05.938254+0000","last_fullsized":"2023-08-25T02:25:05.938254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:41:00.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473432+0000","last_change":"2023-08-25T02:25:04.225050+0000","last_active":"2023-08-25T02:25:06.473432+0000","last_peered":"2023-08-25T02:25:06.473432+0000","last_clean":"2023-08-25T02:25:06.473432+0000","last_became_active":"2023-08-25T02:25:04.224720+0000","last_became_peered":"2023-08-25T02:25:04.224720+0000","last_unstale":"2023-08-25T02:25:06.473432+0000","last_undegraded":"2023-08-25T02:25:06.473432+0000","last_fullsized":"2023-08-25T02:25:06.473432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T08:05:00.583742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214172+0000","last_change":"2023-08-25T02:25:04.228996+0000","last_active":"2023-08-25T02:25:05.214172+0000","last_peered":"2023-08-25T02:25:05.214172+0000","last_clean":"2023-08-25T02:25:05.214172+0000","last_became_active":"2023-08-25T02:25:04.228526+0000","last_became_peered":"2023-08-25T02:25:04.228526+0000","last_unstale":"2023-08-25T02:25:05.214172+0000","last_undegraded":"2023-08-25T02:25:05.214172+0000","last_fullsized":"2023-08-25T02:25:05.214172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:08:48.971024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473735+0000","last_change":"2023-08-25T02:25:04.234921+0000","last_active":"2023-08-25T02:25:06.473735+0000","last_peered":"2023-08-25T02:25:06.473735+0000","last_clean":"2023-08-25T02:25:06.473735+0000","last_became_active":"2023-08-25T02:25:04.234588+0000","last_became_peered":"2023-08-25T02:25:04.234588+0000","last_unstale":"2023-08-25T02:25:06.473735+0000","last_undegraded":"2023-08-25T02:25:06.473735+0000","last_fullsized":"2023-08-25T02:25:06.473735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:21:48.506843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214180+0000","last_change":"2023-08-25T02:25:04.227310+0000","last_active":"2023-08-25T02:25:05.214180+0000","last_peered":"2023-08-25T02:25:05.214180+0000","last_clean":"2023-08-25T02:25:05.214180+0000","last_became_active":"2023-08-25T02:25:04.226940+0000","last_became_peered":"2023-08-25T02:25:04.226940+0000","last_unstale":"2023-08-25T02:25:05.214180+0000","last_undegraded":"2023-08-25T02:25:05.214180+0000","last_fullsized":"2023-08-25T02:25:05.214180+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:21:22.991356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.156816+0000","last_change":"2023-08-25T02:25:04.230835+0000","last_active":"2023-08-25T02:25:06.156816+0000","last_peered":"2023-08-25T02:25:06.156816+0000","last_clean":"2023-08-25T02:25:06.156816+0000","last_became_active":"2023-08-25T02:25:04.230534+0000","last_became_peered":"2023-08-25T02:25:04.230534+0000","last_unstale":"2023-08-25T02:25:06.156816+0000","last_undegraded":"2023-08-25T02:25:06.156816+0000","last_fullsized":"2023-08-25T02:25:06.156816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T07:07:09.527955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.213887+0000","last_change":"2023-08-25T02:25:04.224250+0000","last_active":"2023-08-25T02:25:05.213887+0000","last_peered":"2023-08-25T02:25:05.213887+0000","last_clean":"2023-08-25T02:25:05.213887+0000","last_became_active":"2023-08-25T02:25:04.223591+0000","last_became_peered":"2023-08-25T02:25:04.223591+0000","last_unstale":"2023-08-25T02:25:05.213887+0000","last_undegraded":"2023-08-25T02:25:05.213887+0000","last_fullsized":"2023-08-25T02:25:05.213887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:26:52.465008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.438356+0000","last_change":"2023-08-25T02:25:04.229455+0000","last_active":"2023-08-25T02:25:06.438356+0000","last_peered":"2023-08-25T02:25:06.438356+0000","last_clean":"2023-08-25T02:25:06.438356+0000","last_became_active":"2023-08-25T02:25:04.228114+0000","last_became_peered":"2023-08-25T02:25:04.228114+0000","last_unstale":"2023-08-25T02:25:06.438356+0000","last_undegraded":"2023-08-25T02:25:06.438356+0000","last_fullsized":"2023-08-25T02:25:06.438356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:12:29.745750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:06.473576+0000","last_change":"2023-08-25T02:25:04.226625+0000","last_active":"2023-08-25T02:25:06.473576+0000","last_peered":"2023-08-25T02:25:06.473576+0000","last_clean":"2023-08-25T02:25:06.473576+0000","last_became_active":"2023-08-25T02:25:04.226253+0000","last_became_peered":"2023-08-25T02:25:04.226253+0000","last_unstale":"2023-08-25T02:25:06.473576+0000","last_undegraded":"2023-08-25T02:25:06.473576+0000","last_fullsized":"2023-08-25T02:25:06.473576+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T05:59:23.362440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-08-25T02:25:05.214063+0000","last_change":"2023-08-25T02:25:04.223776+0000","last_active":"2023-08-25T02:25:05.214063+0000","last_peered":"2023-08-25T02:25:05.214063+0000","last_clean":"2023-08-25T02:25:05.214063+0000","last_became_active":"2023-08-25T02:25:04.223597+0000","last_became_peered":"2023-08-25T02:25:04.223597+0000","last_unstale":"2023-08-25T02:25:05.214063+0000","last_undegraded":"2023-08-25T02:25:05.214063+0000","last_fullsized":"2023-08-25T02:25:05.214063+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T02:25:03.593180+0000","last_clean_scrub_stamp":"2023-08-25T02:25:03.593180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T04:26:39.163992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":235,"num_read_kb":131,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":15666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547220,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":411769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.67800000000000005}]},{"osd":2,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.63}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.84399999999999997}]},{"osd":10,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.69099999999999995}]}]},{"osd":10,"up_from":68,"seq":292057776152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1004745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.73499999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.91600000000000004}]},{"osd":3,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.98499999999999999}]},{"osd":6,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.84899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.87},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":1.044}]},{"osd":9,"last update":"Fri Aug 25 02:25:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.88300000000000001}]},{"osd":11,"last update":"Fri Aug 25 02:25:26 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":1.0009999999999999}]}]},{"osd":9,"up_from":63,"seq":270582939677,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":412005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.42999999999999999,"15min":0.439},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.47999999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.45000000000000001,"15min":0.46100000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.501}]},{"osd":1,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.61899999999999999,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.752,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.72799999999999998}]},{"osd":2,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.70299999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.621,"15min":0.63},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.628}]},{"osd":3,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67600000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.56599999999999995,"15min":0.57999999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68799999999999994,"5min":1.153,"15min":1.153},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76900000000000002,"5min":1.103,"15min":1.103},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.66800000000000004,"15min":0.68200000000000005},"min":{"1min":0.437,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.872,"5min":1.2,"15min":1.2},"last":0.80700000000000005}]},{"osd":5,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83199999999999996,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71999999999999997,"15min":0.73199999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85399999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.68100000000000005,"15min":0.70199999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.89000000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77900000000000003,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84899999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91400000000000003,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.871}]},{"osd":8,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50800000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.82899999999999996,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.75600000000000001,"15min":0.77200000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88300000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.70199999999999996}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.82899999999999996}]}]},{"osd":8,"up_from":56,"seq":240518168608,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":411925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.498}]},{"osd":1,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82999999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.86499999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66600000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.433,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.107,"15min":1.107},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.627,"15min":0.626},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78100000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.77000000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74099999999999999,"15min":0.73799999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.71299999999999997}]},{"osd":7,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.74299999999999999,"15min":0.72999999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.53100000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.0489999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.82699999999999996}]},{"osd":9,"last update":"Fri Aug 25 02:25:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.745,"15min":0.73099999999999998},"min":{"1min":0.58699999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.022,"5min":0.78900000000000003,"15min":0.75},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.9750000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.85599999999999998}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.78300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":7,"up_from":49,"seq":210453397541,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":1003347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.497}]},{"osd":1,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42599999999999999,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.44600000000000001,"15min":0.432},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60199999999999998,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.73599999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.65200000000000002,"15min":0.64100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.92100000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.68899999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.70499999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.67100000000000004}]},{"osd":8,"last update":"Fri Aug 25 02:25:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.73199999999999998,"15min":0.71699999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.72299999999999998,"15min":0.70899999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.89300000000000002}]},{"osd":9,"last update":"Fri Aug 25 02:24:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.90600000000000003}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561066,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":860160,"data_stored":411277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.38600000000000001},"min":{"1min":0.28999999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57299999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68300000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68799999999999994}]},{"osd":2,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":3.5659999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.45200000000000001},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5910000000000002,"5min":3.5910000000000002,"15min":3.5910000000000002},"last":0.496}]},{"osd":4,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.76200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.628,"15min":0.628},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79700000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.81499999999999995}]},{"osd":5,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83099999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.46800000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.70399999999999996}]},{"osd":8,"last update":"Fri Aug 25 02:25:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.52900000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.74099999999999999}]},{"osd":9,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":0.67300000000000004},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.72999999999999998}]},{"osd":10,"last update":"Fri Aug 25 02:25:01 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.84199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":412526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.5},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38900000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.55800000000000005,"15min":0.504},"min":{"1min":0.42699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.53000000000000003,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Aug 25 02:25:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55300000000000005,"15min":0.51500000000000001},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78000000000000003,"5min":0.997,"15min":0.997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53400000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67000000000000004,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.371}]},{"osd":4,"last update":"Fri Aug 25 02:24:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.39300000000000002},"min":{"1min":0.33600000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.622},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.45900000000000002,"15min":0.41999999999999998},"min":{"1min":0.36499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.52400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60099999999999998,"15min":0.57899999999999996},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Aug 25 02:25:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.64900000000000002,"15min":0.61899999999999999},"min":{"1min":0.54100000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60299999999999998,"15min":0.58099999999999996},"min":{"1min":0.51000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47299999999999998}]},{"osd":7,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.72499999999999998,"15min":0.70399999999999996},"min":{"1min":0.54400000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.55100000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64000000000000001}]},{"osd":9,"last update":"Fri Aug 25 02:25:45 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.73499999999999999},"min":{"1min":0.53900000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95299999999999996,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.80100000000000005,"15min":0.79200000000000004},"min":{"1min":0.69699999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.044,"5min":1.155,"15min":1.155},"last":0.80400000000000005}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":412665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.434,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.42999999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72999999999999998,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.50600000000000001},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63700000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Aug 25 02:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.45300000000000001,"15min":0.439},"min":{"1min":0.33900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.436,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70299999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.42799999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.626}]},{"osd":6,"last update":"Fri Aug 25 02:25:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69699999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.59099999999999997,"5min":0.754,"15min":0.754},"last":0.33100000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83899999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78500000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70099999999999996}]},{"osd":9,"last update":"Fri Aug 25 02:25:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.623,"15min":0.64000000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68500000000000005,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.39900000000000002}]},{"osd":10,"last update":"Fri Aug 25 02:25:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.54200000000000004}]},{"osd":11,"last update":"Fri Aug 25 02:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":414149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51600000000000001,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.496},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50600000000000001,"15min":0.46700000000000003},"min":{"1min":0.377,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79400000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:25:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.54000000000000004},"min":{"1min":0.46100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.77400000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80800000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Aug 25 02:25:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56399999999999995,"15min":0.52200000000000002},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80600000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58199999999999996,"15min":0.54200000000000004},"min":{"1min":0.504,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81000000000000005,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Aug 25 02:24:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76800000000000002,"5min":0.871,"15min":0.871},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.44800000000000001,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.32700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65000000000000002}]},{"osd":6,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.66100000000000003,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.621},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.58799999999999997}]},{"osd":8,"last update":"Fri Aug 25 02:25:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.746}]},{"osd":9,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.54500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.59999999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93400000000000005,"5min":0.997,"15min":0.997},"last":0.66900000000000004}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":411471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.35199999999999998,"15min":0.32000000000000001},"min":{"1min":0.27900000000000003,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.439},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.26400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Aug 25 02:25:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.52200000000000002},"min":{"1min":0.38,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Aug 25 02:25:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57899999999999996,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.206,"5min":1.252,"15min":1.252},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56599999999999995,"15min":0.53200000000000003},"min":{"1min":0.433,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.55600000000000005,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.63},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Aug 25 02:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.69499999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Aug 25 02:25:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.60699999999999998,"15min":0.52500000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":12.946999999999999,"5min":12.946999999999999,"15min":12.946999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.123,"5min":0.65300000000000002,"15min":0.56499999999999995},"min":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":13.004,"5min":13.004,"15min":13.004},"last":0.40500000000000003}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59599999999999997,"15min":0.59499999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.52600000000000002}]},{"osd":8,"last update":"Fri Aug 25 02:24:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:25:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.35599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.43099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.66800000000000004}]},{"osd":10,"last update":"Fri Aug 25 02:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.65400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":411831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.505,"15min":0.48899999999999999},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.309,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60699999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.38300000000000001,"15min":0.36099999999999999},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Aug 25 02:24:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.44700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.52700000000000002,"15min":0.50900000000000001},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Aug 25 02:25:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.46600000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.442,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Aug 25 02:25:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.61499999999999999,"15min":0.58399999999999996},"min":{"1min":0.49299999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80800000000000005,"5min":1.097,"15min":1.097},"last":0.752},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81699999999999995,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Aug 25 02:24:58 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56599999999999995}]},{"osd":8,"last update":"Fri Aug 25 02:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64200000000000002,"15min":0.63600000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97199999999999998,"5min":1.113,"15min":1.113},"last":0.60299999999999998}]},{"osd":9,"last update":"Fri Aug 25 02:25:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89100000000000001,"5min":1.131,"15min":1.131},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67500000000000004,"15min":0.66600000000000004},"min":{"1min":0.57799999999999996,"5min":0.373,"15min":0.373},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73999999999999999}]},{"osd":10,"last update":"Fri Aug 25 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64400000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1474560,"data_stored":1002700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.745,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.73899999999999999}]},{"osd":2,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.30499999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.42299999999999999,"15min":0.41199999999999998},"min":{"1min":0.28399999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.622},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57999999999999996}]},{"osd":4,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.8999999999999999,"5min":3.8999999999999999,"15min":3.8999999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.46300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81599999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.621,"15min":0.60999999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Aug 25 02:25:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.442,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.748},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.67600000000000005,"15min":0.65100000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.72499999999999998}]},{"osd":8,"last update":"Fri Aug 25 02:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.58199999999999996,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69599999999999995}]},{"osd":9,"last update":"Fri Aug 25 02:25:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69599999999999995,"15min":0.68400000000000005},"min":{"1min":0.51100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72899999999999998,"15min":0.71699999999999997},"min":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76500000000000001}]},{"osd":10,"last update":"Fri Aug 25 02:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T02:25:49.412 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-08-25T02:25:49.412 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-08-25T02:25:49.412 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-08-25T02:25:49.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph health --format=json 2023-08-25T02:25:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:50.478 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:50 smithi089 ceph-mon[98732]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:50.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:50 smithi089 ceph-mon[98732]: pgmap v273: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:50.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:50 smithi121 ceph-mon[100099]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:50.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:50 smithi121 ceph-mon[100099]: pgmap v273: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:50 smithi122 ceph-mon[100457]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-08-25T02:25:50.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:50 smithi122 ceph-mon[100457]: pgmap v273: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:51.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- 172.21.15.89:0/97526836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15980ffc10 msgr2=0x7f15980ffff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 --2- 172.21.15.89:0/97526836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15980ffc10 0x7f15980ffff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1594009990 tx=0x7f1594009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- 172.21.15.89:0/97526836 shutdown_connections 2023-08-25T02:25:51.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 --2- 172.21.15.89:0/97526836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101d10 0x7f159810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 --2- 172.21.15.89:0/97526836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15980ffc10 0x7f15980ffff0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 --2- 172.21.15.89:0/97526836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1598101350 0x7f15981017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- 172.21.15.89:0/97526836 >> 172.21.15.89:0/97526836 conn(0x7f1598076910 msgr2=0x7f1598076d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:51.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- 172.21.15.89:0/97526836 shutdown_connections 2023-08-25T02:25:51.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- 172.21.15.89:0/97526836 wait complete. 2023-08-25T02:25:51.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 Processor -- start 2023-08-25T02:25:51.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.171+0000 7f159f983700 1 -- start start 2023-08-25T02:25:51.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15980ffc10 0x7f15980fc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101350 0x7f15980fcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981a5020 con 0x7f1598101d10 2023-08-25T02:25:51.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981a5190 con 0x7f15980ffc10 2023-08-25T02:25:51.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981a5350 con 0x7f1598101350 2023-08-25T02:25:51.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47230/0 (socket says 172.21.15.89:47230) 2023-08-25T02:25:51.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 -- 172.21.15.89:0/3399116253 learned_addr learned my addr 172.21.15.89:0/3399116253 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:51.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159d71f700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15980ffc10 0x7f15980fc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159cf1e700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101350 0x7f15980fcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 -- 172.21.15.89:0/3399116253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101350 msgr2=0x7f15980fcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101350 0x7f15980fcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 -- 172.21.15.89:0/3399116253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15980ffc10 msgr2=0x7f15980fc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15980ffc10 0x7f15980fc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 -- 172.21.15.89:0/3399116253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940095f0 con 0x7f1598101d10 2023-08-25T02:25:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159df20700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1590007ee0 tx=0x7f159000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590017070 con 0x7f1598101d10 2023-08-25T02:25:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159000f040 con 0x7f1598101d10 2023-08-25T02:25:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590005580 con 0x7f1598101d10 2023-08-25T02:25:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15981a5630 con 0x7f1598101d10 2023-08-25T02:25:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.172+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981a5b30 con 0x7f1598101d10 2023-08-25T02:25:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.173+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598108ef0 con 0x7f1598101d10 2023-08-25T02:25:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.175+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f159000db20 con 0x7f1598101d10 2023-08-25T02:25:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.175+0000 7f158e7fc700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 0x7f15840a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.175+0000 7f159d71f700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 0x7f15840a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.175+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f15900cad90 con 0x7f1598101d10 2023-08-25T02:25:51.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.176+0000 7f159d71f700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 0x7f15840a2490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f159400b5c0 tx=0x7f1594005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:51.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.179+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1590094c70 con 0x7f1598101d10 2023-08-25T02:25:51.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.411+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f159804e620 con 0x7f1598101d10 2023-08-25T02:25:51.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.412+0000 7f158e7fc700 1 -- 172.21.15.89:0/3399116253 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1590098290 con 0x7f1598101d10 2023-08-25T02:25:51.413 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:51.414 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-08-25T02:25:51.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 msgr2=0x7f15840a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 0x7f15840a2490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f159400b5c0 tx=0x7f1594005fd0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 msgr2=0x7f15980fd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1590007ee0 tx=0x7f159000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 shutdown_connections 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f158409ffd0 0x7f15840a2490 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15980ffc10 0x7f15980fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1598101d10 0x7f15980fd370 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 --2- 172.21.15.89:0/3399116253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1598101350 0x7f15980fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 >> 172.21.15.89:0/3399116253 conn(0x7f1598076910 msgr2=0x7f1598068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:51.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 shutdown_connections 2023-08-25T02:25:51.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.414+0000 7f159f983700 1 -- 172.21.15.89:0/3399116253 wait complete. 2023-08-25T02:25:51.721 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-08-25T02:25:51.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 -- 172.21.15.89:0/3562535713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c100930 msgr2=0x7f4e1c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 --2- 172.21.15.89:0/3562535713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c100930 0x7f4e1c100d50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e100099c0 tx=0x7f4e10009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 -- 172.21.15.89:0/3562535713 shutdown_connections 2023-08-25T02:25:51.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 --2- 172.21.15.89:0/3562535713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 --2- 172.21.15.89:0/3562535713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c100930 0x7f4e1c100d50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 --2- 172.21.15.89:0/3562535713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c101b20 0x7f4e1c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 -- 172.21.15.89:0/3562535713 >> 172.21.15.89:0/3562535713 conn(0x7f4e1c0fbe60 msgr2=0x7f4e1c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 -- 172.21.15.89:0/3562535713 shutdown_connections 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.847+0000 7f4e20d66700 1 -- 172.21.15.89:0/3562535713 wait complete. 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 Processor -- start 2023-08-25T02:25:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 -- start start 2023-08-25T02:25:51.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 0x7f4e1c1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c101b20 0x7f4e1c196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19e170 con 0x7f4e1c1024c0 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19e2b0 con 0x7f4e1c100930 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e20d66700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19e440 con 0x7f4e1c101b20 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47244/0 (socket says 172.21.15.89:47244) 2023-08-25T02:25:51.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 -- 172.21.15.89:0/4216455659 learned_addr learned my addr 172.21.15.89:0/4216455659 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1a59c700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 0x7f4e1c1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e19d9b700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c101b20 0x7f4e1c196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 -- 172.21.15.89:0/4216455659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c101b20 msgr2=0x7f4e1c196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c101b20 0x7f4e1c196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 -- 172.21.15.89:0/4216455659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 msgr2=0x7f4e1c1965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.848+0000 7f4e1ad9d700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 0x7f4e1c1965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e1ad9d700 1 -- 172.21.15.89:0/4216455659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100095f0 con 0x7f4e1c1024c0 2023-08-25T02:25:51.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e1a59c700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 0x7f4e1c1965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e1ad9d700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400ef30 tx=0x7f4e1400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14018070 con 0x7f4e1c1024c0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1400f040 con 0x7f4e1c1024c0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14003dc0 con 0x7f4e1c1024c0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c19e6c0 con 0x7f4e1c1024c0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.849+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c19eb80 con 0x7f4e1c1024c0 2023-08-25T02:25:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.850+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1c107860 con 0x7f4e1c1024c0 2023-08-25T02:25:51.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.851+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f4e1400caa0 con 0x7f4e1c1024c0 2023-08-25T02:25:51.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.851+0000 7f4e0b7fe700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 0x7f4e040a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:51.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.852+0000 7f4e1a59c700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 0x7f4e040a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:51.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.852+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4e140cb6c0 con 0x7f4e1c1024c0 2023-08-25T02:25:51.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.852+0000 7f4e1a59c700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 0x7f4e040a24e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009990 tx=0x7f4e1001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:51.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:51.855+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4e14095630 con 0x7f4e1c1024c0 2023-08-25T02:25:52.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.048+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e1c10c3e0 con 0x7f4e1c1024c0 2023-08-25T02:25:52.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.050+0000 7f4e0b7fe700 1 -- 172.21.15.89:0/4216455659 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f4e14098c50 con 0x7f4e1c1024c0 2023-08-25T02:25:52.051 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:52.052 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:52.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 msgr2=0x7f4e040a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 0x7f4e040a24e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009990 tx=0x7f4e1001a040 comp rx=0 tx=0).stop 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 msgr2=0x7f4e1c19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400ef30 tx=0x7f4e1400c450 comp rx=0 tx=0).stop 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 shutdown_connections 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f4e040a0020 0x7f4e040a24e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e1c100930 0x7f4e1c1965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e1c1024c0 0x7f4e1c19dba0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 --2- 172.21.15.89:0/4216455659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e1c101b20 0x7f4e1c196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 >> 172.21.15.89:0/4216455659 conn(0x7f4e1c0fbe60 msgr2=0x7f4e1c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 shutdown_connections 2023-08-25T02:25:52.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.052+0000 7f4e20d66700 1 -- 172.21.15.89:0/4216455659 wait complete. 2023-08-25T02:25:52.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:25:52.067 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:25:52.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.191+0000 7f0bc5553700 1 -- 172.21.15.89:0/2252389802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 msgr2=0x7f0bc0107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.191+0000 7f0bc5553700 1 --2- 172.21.15.89:0/2252389802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc0107c50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00099c0 tx=0x7f0bb0009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 -- 172.21.15.89:0/2252389802 shutdown_connections 2023-08-25T02:25:52.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 --2- 172.21.15.89:0/2252389802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc0107c50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 --2- 172.21.15.89:0/2252389802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc00ff810 0x7f0bc00ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 --2- 172.21.15.89:0/2252389802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0100a00 0x7f0bc0100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 -- 172.21.15.89:0/2252389802 >> 172.21.15.89:0/2252389802 conn(0x7f0bc00fad80 msgr2=0x7f0bc00fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 -- 172.21.15.89:0/2252389802 shutdown_connections 2023-08-25T02:25:52.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 -- 172.21.15.89:0/2252389802 wait complete. 2023-08-25T02:25:52.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 Processor -- start 2023-08-25T02:25:52.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.192+0000 7f0bc5553700 1 -- start start 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 0x7f0bc01965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc0100a00 0x7f0bc0196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc019e120 con 0x7f0bc01013a0 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc019e260 con 0x7f0bc00ff810 2023-08-25T02:25:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc019e3f0 con 0x7f0bc0100a00 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47258/0 (socket says 172.21.15.89:47258) 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 -- 172.21.15.89:0/1535316087 learned_addr learned my addr 172.21.15.89:0/1535316087 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbe7fc700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc0100a00 0x7f0bc0196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbeffd700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 0x7f0bc01965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 -- 172.21.15.89:0/1535316087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc0100a00 msgr2=0x7f0bc0196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc0100a00 0x7f0bc0196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 -- 172.21.15.89:0/1535316087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 msgr2=0x7f0bc01965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 0x7f0bc01965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 -- 172.21.15.89:0/1535316087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00095f0 con 0x7f0bc01013a0 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbeffd700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 0x7f0bc01965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bbf7fe700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006010 tx=0x7f0bb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb001d070 con 0x7f0bc01013a0 2023-08-25T02:25:52.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb0003f90 con 0x7f0bc01013a0 2023-08-25T02:25:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc019e670 con 0x7f0bc01013a0 2023-08-25T02:25:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0022d70 con 0x7f0bc01013a0 2023-08-25T02:25:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.193+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc019ea90 con 0x7f0bc01013a0 2023-08-25T02:25:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.194+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0107840 con 0x7f0bc01013a0 2023-08-25T02:25:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.195+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0bb0022470 con 0x7f0bc01013a0 2023-08-25T02:25:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.196+0000 7f0ba7fff700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 0x7f0ba80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.196+0000 7f0bbeffd700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 0x7f0ba80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.196+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0bb00ccc90 con 0x7f0bc01013a0 2023-08-25T02:25:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.196+0000 7f0bbeffd700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 0x7f0ba80a2410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0100860 tx=0x7f0bb8007c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.199+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0bb0096c00 con 0x7f0bc01013a0 2023-08-25T02:25:52.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.395+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bc002cc20 con 0x7f0bc01013a0 2023-08-25T02:25:52.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.396+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1535316087 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bb009a220 con 0x7f0bc01013a0 2023-08-25T02:25:52.397 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:25:52.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 msgr2=0x7f0ba80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 0x7f0ba80a2410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0100860 tx=0x7f0bb8007c20 comp rx=0 tx=0).stop 2023-08-25T02:25:52.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 msgr2=0x7f0bc019db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006010 tx=0x7f0bb0004730 comp rx=0 tx=0).stop 2023-08-25T02:25:52.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 shutdown_connections 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0ba809ff50 0x7f0ba80a2410 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc00ff810 0x7f0bc01965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bc01013a0 0x7f0bc019db50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 --2- 172.21.15.89:0/1535316087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0bc0100a00 0x7f0bc0196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 >> 172.21.15.89:0/1535316087 conn(0x7f0bc00fad80 msgr2=0x7f0bc00fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 shutdown_connections 2023-08-25T02:25:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.398+0000 7f0bc5553700 1 -- 172.21.15.89:0/1535316087 wait complete. 2023-08-25T02:25:52.412 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:25:52.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 -- 172.21.15.89:0/346075511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 msgr2=0x7faf84107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/346075511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf84107c90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00b0d0 tx=0x7faf7c00b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 -- 172.21.15.89:0/346075511 shutdown_connections 2023-08-25T02:25:52.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/346075511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf840ff850 0x7faf840ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/346075511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf84107c90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/346075511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf84100a40 0x7faf84100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 -- 172.21.15.89:0/346075511 >> 172.21.15.89:0/346075511 conn(0x7faf840fada0 msgr2=0x7faf840fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 -- 172.21.15.89:0/346075511 shutdown_connections 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.534+0000 7faf8aa3f700 1 -- 172.21.15.89:0/346075511 wait complete. 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 Processor -- start 2023-08-25T02:25:52.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 -- start start 2023-08-25T02:25:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf840ff850 0x7faf84196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf84100a40 0x7faf84196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8419e380 con 0x7faf841013e0 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8419e4c0 con 0x7faf84100a40 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf8aa3f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8419e650 con 0x7faf840ff850 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf88fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf88fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47272/0 (socket says 172.21.15.89:47272) 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf88fdc700 1 -- 172.21.15.89:0/2159296309 learned_addr learned my addr 172.21.15.89:0/2159296309 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.535+0000 7faf83fff700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf840ff850 0x7faf84196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf837fe700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf84100a40 0x7faf84196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 -- 172.21.15.89:0/2159296309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf840ff850 msgr2=0x7faf84196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf840ff850 0x7faf84196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 -- 172.21.15.89:0/2159296309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf84100a40 msgr2=0x7faf84196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf84100a40 0x7faf84196d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 -- 172.21.15.89:0/2159296309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c009d00 con 0x7faf841013e0 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf88fdc700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009300 tx=0x7faf7c009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c00e040 con 0x7faf841013e0 2023-08-25T02:25:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf7c00ceb0 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8419e8d0 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c01b740 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.536+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8419edd0 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.537+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8410b130 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.538+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7faf7c005050 con 0x7faf841013e0 2023-08-25T02:25:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.538+0000 7faf817fa700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 0x7faf6c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.539+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faf7c0ccbf0 con 0x7faf841013e0 2023-08-25T02:25:52.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.539+0000 7faf83fff700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 0x7faf6c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.539+0000 7faf83fff700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 0x7faf6c0a2410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faf841008a0 tx=0x7faf7400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.542+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7faf7c096b60 con 0x7faf841013e0 2023-08-25T02:25:52.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:52 smithi121 ceph-mon[100099]: pgmap v274: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:52 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3399116253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-08-25T02:25:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:52 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4216455659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:52 smithi122 ceph-mon[100457]: pgmap v274: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:52 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3399116253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-08-25T02:25:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:52 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4216455659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:52.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.731+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf8419eff0 con 0x7faf841013e0 2023-08-25T02:25:52.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.732+0000 7faf817fa700 1 -- 172.21.15.89:0/2159296309 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf7c004eb0 con 0x7faf841013e0 2023-08-25T02:25:52.733 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:25:52.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 msgr2=0x7faf6c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 0x7faf6c0a2410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faf841008a0 tx=0x7faf7400a380 comp rx=0 tx=0).stop 2023-08-25T02:25:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 msgr2=0x7faf8419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009300 tx=0x7faf7c009330 comp rx=0 tx=0).stop 2023-08-25T02:25:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 shutdown_connections 2023-08-25T02:25:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf841013e0 0x7faf8419ddb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faf840ff850 0x7faf84196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7faf6c09ff50 0x7faf6c0a2410 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 --2- 172.21.15.89:0/2159296309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf84100a40 0x7faf84196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 >> 172.21.15.89:0/2159296309 conn(0x7faf840fada0 msgr2=0x7faf840fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 shutdown_connections 2023-08-25T02:25:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.734+0000 7faf8aa3f700 1 -- 172.21.15.89:0/2159296309 wait complete. 2023-08-25T02:25:52.746 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:52 smithi089 ceph-mon[98732]: pgmap v274: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:52.746 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:52 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3399116253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-08-25T02:25:52.747 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:52 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4216455659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:52.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:25:52.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 -- 172.21.15.89:0/1345330413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c0ff810 msgr2=0x7fde4c0ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 --2- 172.21.15.89:0/1345330413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c0ff810 0x7fde4c0ffc30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0099c0 tx=0x7fde3c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 -- 172.21.15.89:0/1345330413 shutdown_connections 2023-08-25T02:25:52.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 --2- 172.21.15.89:0/1345330413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c1013a0 0x7fde4c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 --2- 172.21.15.89:0/1345330413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c0ff810 0x7fde4c0ffc30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 --2- 172.21.15.89:0/1345330413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c100a00 0x7fde4c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 -- 172.21.15.89:0/1345330413 >> 172.21.15.89:0/1345330413 conn(0x7fde4c0fad80 msgr2=0x7fde4c0fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:52.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 -- 172.21.15.89:0/1345330413 shutdown_connections 2023-08-25T02:25:52.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.870+0000 7fde547f1700 1 -- 172.21.15.89:0/1345330413 wait complete. 2023-08-25T02:25:52.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 Processor -- start 2023-08-25T02:25:52.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 -- start start 2023-08-25T02:25:52.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c0ff810 0x7fde4c1955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c1013a0 0x7fde4c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c19d170 con 0x7fde4c100a00 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c19d2b0 con 0x7fde4c0ff810 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde547f1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c19d440 con 0x7fde4c1013a0 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47284/0 (socket says 172.21.15.89:47284) 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 -- 172.21.15.89:0/3425596366 learned_addr learned my addr 172.21.15.89:0/3425596366 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:52.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 -- 172.21.15.89:0/3425596366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c1013a0 msgr2=0x7fde4c19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c1013a0 0x7fde4c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 -- 172.21.15.89:0/3425596366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c0ff810 msgr2=0x7fde4c1955f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde5258d700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c0ff810 0x7fde4c1955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.871+0000 7fde51d8c700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c0ff810 0x7fde4c1955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde51d8c700 1 -- 172.21.15.89:0/3425596366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c0095f0 con 0x7fde4c100a00 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde51d8c700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fde4800d950 tx=0x7fde4800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48009b40 con 0x7fde4c100a00 2023-08-25T02:25:52.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde48009ca0 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c19d720 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4800f620 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.872+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c19dc30 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.873+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c106ea0 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.874+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fde48010460 con 0x7fde4c100a00 2023-08-25T02:25:52.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.874+0000 7fde437fe700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 0x7fde380a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:52.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.875+0000 7fde5258d700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 0x7fde380a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:52.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.875+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde480cb510 con 0x7fde4c100a00 2023-08-25T02:25:52.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.875+0000 7fde5258d700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 0x7fde380a2410 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098e0 tx=0x7fde3c005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:52.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:52.878+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fde48095350 con 0x7fde4c100a00 2023-08-25T02:25:53.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.069+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde4c02cc20 con 0x7fde4c100a00 2023-08-25T02:25:53.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.069+0000 7fde437fe700 1 -- 172.21.15.89:0/3425596366 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde48020020 con 0x7fde4c100a00 2023-08-25T02:25:53.070 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:25:53.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 msgr2=0x7fde380a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 0x7fde380a2410 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098e0 tx=0x7fde3c005c00 comp rx=0 tx=0).stop 2023-08-25T02:25:53.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 msgr2=0x7fde4c195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fde4800d950 tx=0x7fde4800dd10 comp rx=0 tx=0).stop 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 shutdown_connections 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fde3809ff50 0x7fde380a2410 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde4c0ff810 0x7fde4c1955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c1013a0 0x7fde4c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 --2- 172.21.15.89:0/3425596366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde4c100a00 0x7fde4c195b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 >> 172.21.15.89:0/3425596366 conn(0x7fde4c0fad80 msgr2=0x7fde4c103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 shutdown_connections 2023-08-25T02:25:53.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.071+0000 7fde547f1700 1 -- 172.21.15.89:0/3425596366 wait complete. 2023-08-25T02:25:53.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:25:53.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.204+0000 7f0178d65700 1 -- 172.21.15.89:0/1335753410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 msgr2=0x7f0174101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.204+0000 7f0178d65700 1 --2- 172.21.15.89:0/1335753410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174101f70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f01680099c0 tx=0x7f0168009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- 172.21.15.89:0/1335753410 shutdown_connections 2023-08-25T02:25:53.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- 172.21.15.89:0/1335753410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741024b0 0x7f0174108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- 172.21.15.89:0/1335753410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0174100920 0x7f0174100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- 172.21.15.89:0/1335753410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174101f70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- 172.21.15.89:0/1335753410 >> 172.21.15.89:0/1335753410 conn(0x7f01740fbe50 msgr2=0x7f01740fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- 172.21.15.89:0/1335753410 shutdown_connections 2023-08-25T02:25:53.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- 172.21.15.89:0/1335753410 wait complete. 2023-08-25T02:25:53.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 Processor -- start 2023-08-25T02:25:53.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- start start 2023-08-25T02:25:53.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 0x7f0174196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01741024b0 0x7f017419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017419e190 con 0x7f0174101b10 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017419e2d0 con 0x7f0174100920 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.205+0000 7f0178d65700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017419e460 con 0x7f01741024b0 2023-08-25T02:25:53.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47296/0 (socket says 172.21.15.89:47296) 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 -- 172.21.15.89:0/2401050039 learned_addr learned my addr 172.21.15.89:0/2401050039 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0172d9d700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01741024b0 0x7f017419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f017259c700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 0x7f0174196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 -- 172.21.15.89:0/2401050039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01741024b0 msgr2=0x7f017419dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01741024b0 0x7f017419dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 -- 172.21.15.89:0/2401050039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 msgr2=0x7f0174196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 0x7f0174196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 -- 172.21.15.89:0/2401050039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01680095f0 con 0x7f0174101b10 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f017259c700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 0x7f0174196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0171d9b700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016801d070 con 0x7f0174101b10 2023-08-25T02:25:53.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01680040e0 con 0x7f0174101b10 2023-08-25T02:25:53.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017419e6e0 con 0x7f0174101b10 2023-08-25T02:25:53.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800f460 con 0x7f0174101b10 2023-08-25T02:25:53.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.206+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017419eb00 con 0x7f0174101b10 2023-08-25T02:25:53.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.207+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174107850 con 0x7f0174101b10 2023-08-25T02:25:53.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.208+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0168022570 con 0x7f0174101b10 2023-08-25T02:25:53.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.208+0000 7f01637fe700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 0x7f015c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.209+0000 7f017259c700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 0x7f015c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.209+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f01680ccc00 con 0x7f0174101b10 2023-08-25T02:25:53.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.209+0000 7f017259c700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 0x7f015c0a2460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0174101970 tx=0x7f016400b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.212+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0168096b70 con 0x7f0174101b10 2023-08-25T02:25:53.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.450+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f017402ceb0 con 0x7f0174101b10 2023-08-25T02:25:53.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.451+0000 7f01637fe700 1 -- 172.21.15.89:0/2401050039 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7f01680ade60 con 0x7f0174101b10 2023-08-25T02:25:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:53.453 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:25:53.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 msgr2=0x7f015c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 0x7f015c0a2460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0174101970 tx=0x7f016400b670 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 msgr2=0x7f0174196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168009580 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 shutdown_connections 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f015c09ffa0 0x7f015c0a2460 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0174100920 0x7f0174196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01741024b0 0x7f017419dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 --2- 172.21.15.89:0/2401050039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0174101b10 0x7f0174196b50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 >> 172.21.15.89:0/2401050039 conn(0x7f01740fbe50 msgr2=0x7f01740fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 shutdown_connections 2023-08-25T02:25:53.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.453+0000 7f0178d65700 1 -- 172.21.15.89:0/2401050039 wait complete. 2023-08-25T02:25:53.456 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:25:53.467 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- 172.21.15.89:0/3218823873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe150100a30 msgr2=0x7fe150100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 --2- 172.21.15.89:0/3218823873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe150100a30 0x7fe150100e90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe1400099c0 tx=0x7fe140009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- 172.21.15.89:0/3218823873 shutdown_connections 2023-08-25T02:25:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 --2- 172.21.15.89:0/3218823873 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1501013d0 0x7fe150107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 --2- 172.21.15.89:0/3218823873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1500ff840 0x7fe1500ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 --2- 172.21.15.89:0/3218823873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe150100a30 0x7fe150100e90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- 172.21.15.89:0/3218823873 >> 172.21.15.89:0/3218823873 conn(0x7fe1500fadb0 msgr2=0x7fe1500fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- 172.21.15.89:0/3218823873 shutdown_connections 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- 172.21.15.89:0/3218823873 wait complete. 2023-08-25T02:25:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 Processor -- start 2023-08-25T02:25:53.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.587+0000 7fe155952700 1 -- start start 2023-08-25T02:25:53.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1500ff840 0x7fe1501965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe150100a30 0x7fe150196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15019e140 con 0x7fe1501013d0 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15019e280 con 0x7fe1500ff840 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15019e410 con 0x7fe150100a30 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47310/0 (socket says 172.21.15.89:47310) 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 -- 172.21.15.89:0/768782953 learned_addr learned my addr 172.21.15.89:0/768782953 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14effd700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1500ff840 0x7fe1501965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14e7fc700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe150100a30 0x7fe150196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 -- 172.21.15.89:0/768782953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe150100a30 msgr2=0x7fe150196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe150100a30 0x7fe150196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 -- 172.21.15.89:0/768782953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1500ff840 msgr2=0x7fe1501965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1500ff840 0x7fe1501965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 -- 172.21.15.89:0/768782953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1400095f0 con 0x7fe1501013d0 2023-08-25T02:25:53.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe14f7fe700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe14800b970 tx=0x7fe14800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14800d860 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15019e6f0 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe14800de80 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.588+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148017400 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.589+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15019ebb0 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.589+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15010b0d0 con 0x7fe1501013d0 2023-08-25T02:25:53.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.590+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fe14800e630 con 0x7fe1501013d0 2023-08-25T02:25:53.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.591+0000 7fe154950700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 0x7fe13c0a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.591+0000 7fe14effd700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 0x7fe13c0a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.591+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe148006e40 con 0x7fe1501013d0 2023-08-25T02:25:53.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.591+0000 7fe14effd700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 0x7fe13c0a2460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe150100890 tx=0x7fe13800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.595+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fe148094d90 con 0x7fe1501013d0 2023-08-25T02:25:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:53 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1535316087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:53 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2159296309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:53 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3425596366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:53 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1535316087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:53 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2159296309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:53 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3425596366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.788+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe15019ee30 con 0x7fe1501013d0 2023-08-25T02:25:53.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.789+0000 7fe154950700 1 -- 172.21.15.89:0/768782953 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7fe1480983b0 con 0x7fe1501013d0 2023-08-25T02:25:53.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:53.791 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 msgr2=0x7fe13c0a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 0x7fe13c0a2460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe150100890 tx=0x7fe13800a380 comp rx=0 tx=0).stop 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 msgr2=0x7fe15019db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe14800b970 tx=0x7fe14800bc80 comp rx=0 tx=0).stop 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 shutdown_connections 2023-08-25T02:25:53.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe13c09ffa0 0x7fe13c0a2460 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1500ff840 0x7fe1501965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1501013d0 0x7fe15019db70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 --2- 172.21.15.89:0/768782953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe150100a30 0x7fe150196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 >> 172.21.15.89:0/768782953 conn(0x7fe1500fadb0 msgr2=0x7fe150068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 shutdown_connections 2023-08-25T02:25:53.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.792+0000 7fe155952700 1 -- 172.21.15.89:0/768782953 wait complete. 2023-08-25T02:25:53.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:53 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1535316087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:53 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2159296309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:53 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3425596366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:53.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:25:53.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:25:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 -- 172.21.15.89:0/1134477135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 msgr2=0x7ff9501005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 --2- 172.21.15.89:0/1134477135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff9501005d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff94000aad0 tx=0x7ff94000ade0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 -- 172.21.15.89:0/1134477135 shutdown_connections 2023-08-25T02:25:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 --2- 172.21.15.89:0/1134477135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 0x7ff9500ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 --2- 172.21.15.89:0/1134477135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff9501073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 --2- 172.21.15.89:0/1134477135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff9501005d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 -- 172.21.15.89:0/1134477135 >> 172.21.15.89:0/1134477135 conn(0x7ff9500fa4f0 msgr2=0x7ff9500fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:53.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 -- 172.21.15.89:0/1134477135 shutdown_connections 2023-08-25T02:25:53.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.929+0000 7ff955cdb700 1 -- 172.21.15.89:0/1134477135 wait complete. 2023-08-25T02:25:53.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 Processor -- start 2023-08-25T02:25:53.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 -- start start 2023-08-25T02:25:53.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 0x7ff950194400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff95019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95019bf80 con 0x7ff950100170 2023-08-25T02:25:53.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95019c0c0 con 0x7ff9500fef80 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff955cdb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95019c250 con 0x7ff950100b10 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47322/0 (socket says 172.21.15.89:47322) 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 -- 172.21.15.89:0/2822062619 learned_addr learned my addr 172.21.15.89:0/2822062619 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94ffff700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff95019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 -- 172.21.15.89:0/2822062619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 msgr2=0x7ff95019b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff95019b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 -- 172.21.15.89:0/2822062619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 msgr2=0x7ff950194400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94f7fe700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 0x7ff950194400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 0x7ff950194400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 -- 172.21.15.89:0/2822062619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94000a700 con 0x7ff950100170 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94ffff700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff95019b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.930+0000 7ff94effd700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a9f0 tx=0x7ff94000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.931+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940004280 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.931+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95019c4d0 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.931+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff94000e910 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.931+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9400168c0 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.931+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95019c8f0 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.932+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff950109f00 con 0x7ff950100170 2023-08-25T02:25:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.933+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7ff94000ea80 con 0x7ff950100170 2023-08-25T02:25:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.934+0000 7ff94cff9700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 0x7ff9380a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.934+0000 7ff94f7fe700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 0x7ff9380a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.934+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff940012070 con 0x7ff950100170 2023-08-25T02:25:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.934+0000 7ff94f7fe700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 0x7ff9380a2410 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9500fffd0 tx=0x7ff944005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:53.937+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff940098560 con 0x7ff950100170 2023-08-25T02:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.130+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff950105ef0 con 0x7ff950100170 2023-08-25T02:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.130+0000 7ff94cff9700 1 -- 172.21.15.89:0/2822062619 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff94009bb80 con 0x7ff950100170 2023-08-25T02:25:54.131 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 msgr2=0x7ff9380a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 0x7ff9380a2410 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9500fffd0 tx=0x7ff944005cf0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 msgr2=0x7ff950194940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a9f0 tx=0x7ff94000ced0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 shutdown_connections 2023-08-25T02:25:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff93809ff50 0x7ff9380a2410 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9500fef80 0x7ff950194400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff950100b10 0x7ff95019b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 --2- 172.21.15.89:0/2822062619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff950100170 0x7ff950194940 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 >> 172.21.15.89:0/2822062619 conn(0x7ff9500fa4f0 msgr2=0x7ff9500fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 shutdown_connections 2023-08-25T02:25:54.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.132+0000 7ff955cdb700 1 -- 172.21.15.89:0/2822062619 wait complete. 2023-08-25T02:25:54.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:25:54.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 -- 172.21.15.89:0/969406149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9541013d0 msgr2=0x7fe954107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/969406149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9541013d0 0x7fe954107c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00b0d0 tx=0x7fe94c00b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 -- 172.21.15.89:0/969406149 shutdown_connections 2023-08-25T02:25:54.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/969406149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9540ff840 0x7fe9540ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/969406149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9541013d0 0x7fe954107c80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/969406149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe954100a30 0x7fe954100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 -- 172.21.15.89:0/969406149 >> 172.21.15.89:0/969406149 conn(0x7fe9540fadb0 msgr2=0x7fe9540fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 -- 172.21.15.89:0/969406149 shutdown_connections 2023-08-25T02:25:54.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.269+0000 7fe95a5fb700 1 -- 172.21.15.89:0/969406149 wait complete. 2023-08-25T02:25:54.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 Processor -- start 2023-08-25T02:25:54.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 -- start start 2023-08-25T02:25:54.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9540ff840 0x7fe954069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 0x7fe95419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95419e200 con 0x7fe954100a30 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95419e370 con 0x7fe9541013d0 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe95a5fb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95419e530 con 0x7fe9540ff840 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe94bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe94bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47336/0 (socket says 172.21.15.89:47336) 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe94bfff700 1 -- 172.21.15.89:0/2189057773 learned_addr learned my addr 172.21.15.89:0/2189057773 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:54.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.270+0000 7fe958b98700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 0x7fe95419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 -- 172.21.15.89:0/2189057773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9540ff840 msgr2=0x7fe954069620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe953fff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9540ff840 0x7fe954069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9540ff840 0x7fe954069620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 -- 172.21.15.89:0/2189057773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 msgr2=0x7fe95419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 0x7fe95419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 -- 172.21.15.89:0/2189057773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c009d00 con 0x7fe954100a30 2023-08-25T02:25:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe958b98700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 0x7fe95419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe94bfff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00d980 tx=0x7fe93c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00bb60 con 0x7fe954100a30 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe93c007640 con 0x7fe954100a30 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95419e810 con 0x7fe954100a30 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c004e90 con 0x7fe954100a30 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.271+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95419ed20 con 0x7fe954100a30 2023-08-25T02:25:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.272+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9541066f0 con 0x7fe954100a30 2023-08-25T02:25:54.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.273+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fe93c00bcc0 con 0x7fe954100a30 2023-08-25T02:25:54.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.273+0000 7fe951ffb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 0x7fe9400a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.273+0000 7fe953fff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 0x7fe9400a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.273+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe93c0cb4b0 con 0x7fe954100a30 2023-08-25T02:25:54.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.274+0000 7fe953fff700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 0x7fe9400a2410 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe944005950 tx=0x7fe9440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.277+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fe93c098e20 con 0x7fe954100a30 2023-08-25T02:25:54.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.469+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe95410b290 con 0x7fe954100a30 2023-08-25T02:25:54.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.470+0000 7fe951ffb700 1 -- 172.21.15.89:0/2189057773 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe93c016020 con 0x7fe954100a30 2023-08-25T02:25:54.471 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 msgr2=0x7fe9400a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 0x7fe9400a2410 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe944005950 tx=0x7fe9440058e0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 msgr2=0x7fe95419b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00d980 tx=0x7fe93c00dd40 comp rx=0 tx=0).stop 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 shutdown_connections 2023-08-25T02:25:54.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe94009ff50 0x7fe9400a2410 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9541013d0 0x7fe95419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9540ff840 0x7fe954069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 --2- 172.21.15.89:0/2189057773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954100a30 0x7fe95419b3d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 >> 172.21.15.89:0/2189057773 conn(0x7fe9540fadb0 msgr2=0x7fe9540fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 shutdown_connections 2023-08-25T02:25:54.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.472+0000 7fe95a5fb700 1 -- 172.21.15.89:0/2189057773 wait complete. 2023-08-25T02:25:54.484 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:54 smithi089 ceph-mon[98732]: pgmap v275: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:54.484 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:54 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2401050039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.484 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:54 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/768782953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.485 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:54 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2822062619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:54.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:25:54.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- 172.21.15.89:0/2786453521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 msgr2=0x7fc684108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 --2- 172.21.15.89:0/2786453521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684108ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00b0a0 tx=0x7fc67c00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- 172.21.15.89:0/2786453521 shutdown_connections 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 --2- 172.21.15.89:0/2786453521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684108ee0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 --2- 172.21.15.89:0/2786453521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc684101f00 0x7fc684102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 --2- 172.21.15.89:0/2786453521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6840fe970 0x7fc6840fedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- 172.21.15.89:0/2786453521 >> 172.21.15.89:0/2786453521 conn(0x7fc6840fa4f0 msgr2=0x7fc6840fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- 172.21.15.89:0/2786453521 shutdown_connections 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- 172.21.15.89:0/2786453521 wait complete. 2023-08-25T02:25:54.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 Processor -- start 2023-08-25T02:25:54.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.611+0000 7fc68a370700 1 -- start start 2023-08-25T02:25:54.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6840fe970 0x7fc684196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 0x7fc68419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68419e1b0 con 0x7fc6840ff330 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68419e2f0 con 0x7fc684101f00 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68419e480 con 0x7fc6840fe970 2023-08-25T02:25:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47350/0 (socket says 172.21.15.89:47350) 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 -- 172.21.15.89:0/965134812 learned_addr learned my addr 172.21.15.89:0/965134812 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68890d700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 0x7fc68419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 -- 172.21.15.89:0/965134812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6840fe970 msgr2=0x7fc684196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc683fff700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6840fe970 0x7fc684196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6840fe970 0x7fc684196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 -- 172.21.15.89:0/965134812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 msgr2=0x7fc68419dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 0x7fc68419dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 -- 172.21.15.89:0/965134812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c009d00 con 0x7fc6840ff330 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68890d700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 0x7fc68419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6837fe700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc67400e970 tx=0x7fc67400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674009790 con 0x7fc6840ff330 2023-08-25T02:25:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68419e760 con 0x7fc6840ff330 2023-08-25T02:25:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6740044e0 con 0x7fc6840ff330 2023-08-25T02:25:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.612+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674003e30 con 0x7fc6840ff330 2023-08-25T02:25:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.613+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68419ecc0 con 0x7fc6840ff330 2023-08-25T02:25:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.614+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68410c110 con 0x7fc6840ff330 2023-08-25T02:25:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.615+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fc674004000 con 0x7fc6840ff330 2023-08-25T02:25:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.615+0000 7fc6817fa700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 0x7fc66c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.616+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc674014070 con 0x7fc6840ff330 2023-08-25T02:25:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.616+0000 7fc683fff700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 0x7fc66c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.616+0000 7fc683fff700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 0x7fc66c0a2410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc684067780 tx=0x7fc678005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.619+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fc674095430 con 0x7fc6840ff330 2023-08-25T02:25:54.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:54 smithi121 ceph-mon[100099]: pgmap v275: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:54.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:54 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2401050039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:54 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/768782953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:54 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2822062619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:54 smithi122 ceph-mon[100457]: pgmap v275: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:54 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2401050039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:54 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/768782953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:54 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2822062619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:25:54.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.811+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6841078e0 con 0x7fc6840ff330 2023-08-25T02:25:54.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.811+0000 7fc6817fa700 1 -- 172.21.15.89:0/965134812 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc674098a50 con 0x7fc6840ff330 2023-08-25T02:25:54.813 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:25:54.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 msgr2=0x7fc66c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 0x7fc66c0a2410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc684067780 tx=0x7fc678005d70 comp rx=0 tx=0).stop 2023-08-25T02:25:54.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 msgr2=0x7fc684196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc67400e970 tx=0x7fc67400ed30 comp rx=0 tx=0).stop 2023-08-25T02:25:54.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 shutdown_connections 2023-08-25T02:25:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fc66c09ff50 0x7fc66c0a2410 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc684101f00 0x7fc68419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6840fe970 0x7fc684196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 --2- 172.21.15.89:0/965134812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6840ff330 0x7fc684196b70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 >> 172.21.15.89:0/965134812 conn(0x7fc6840fa4f0 msgr2=0x7fc6840fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 shutdown_connections 2023-08-25T02:25:54.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.814+0000 7fc68a370700 1 -- 172.21.15.89:0/965134812 wait complete. 2023-08-25T02:25:54.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:25:54.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1302994679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc81009f0 msgr2=0x7f7cc8100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1302994679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc81009f0 0x7f7cc8100e50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80099c0 tx=0x7f7cb8009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1302994679 shutdown_connections 2023-08-25T02:25:54.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1302994679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8101390 0x7f7cc8107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1302994679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc80ff800 0x7f7cc80ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1302994679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc81009f0 0x7f7cc8100e50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1302994679 >> 172.21.15.89:0/1302994679 conn(0x7f7cc80fad50 msgr2=0x7f7cc80fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:54.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1302994679 shutdown_connections 2023-08-25T02:25:54.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1302994679 wait complete. 2023-08-25T02:25:54.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 Processor -- start 2023-08-25T02:25:54.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.948+0000 7f7ccdae5700 1 -- start start 2023-08-25T02:25:54.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 0x7f7cc819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc8101390 0x7f7cc819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc819e110 con 0x7f7cc80ff800 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc819e280 con 0x7f7cc81009f0 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc819e440 con 0x7f7cc8101390 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47362/0 (socket says 172.21.15.89:47362) 2023-08-25T02:25:54.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 -- 172.21.15.89:0/1574101771 learned_addr learned my addr 172.21.15.89:0/1574101771 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc7fff700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc8101390 0x7f7cc819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc6ffd700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 0x7f7cc819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 -- 172.21.15.89:0/1574101771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc8101390 msgr2=0x7f7cc819d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc8101390 0x7f7cc819d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 -- 172.21.15.89:0/1574101771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 msgr2=0x7f7cc819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 0x7f7cc819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:54.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 -- 172.21.15.89:0/1574101771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80095f0 con 0x7f7cc80ff800 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc6ffd700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 0x7f7cc819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc77fe700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00d980 tx=0x7f7cbc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc00de60 con 0x7f7cc80ff800 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cbc0044e0 con 0x7f7cc80ff800 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc819e720 con 0x7f7cc80ff800 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc003be0 con 0x7f7cc80ff800 2023-08-25T02:25:54.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.949+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc819ec30 con 0x7f7cc80ff800 2023-08-25T02:25:54.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.950+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc8106d90 con 0x7f7cc80ff800 2023-08-25T02:25:54.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.951+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f7cbc009bb0 con 0x7f7cc80ff800 2023-08-25T02:25:54.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.952+0000 7f7cc4ff9700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 0x7f7cb00a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:54.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.952+0000 7f7cc6ffd700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 0x7f7cb00a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:54.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.952+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7cbc0caf20 con 0x7f7cc80ff800 2023-08-25T02:25:54.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.952+0000 7f7cc6ffd700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 0x7f7cb00a24e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006010 tx=0x7f7cb8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:54.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:54.957+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7cbc094d60 con 0x7f7cc80ff800 2023-08-25T02:25:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.193+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cc810b010 con 0x7f7cc80ff800 2023-08-25T02:25:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.194+0000 7f7cc4ff9700 1 -- 172.21.15.89:0/1574101771 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7f7cbc009940 con 0x7f7cc80ff800 2023-08-25T02:25:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:55.195 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:25:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 msgr2=0x7f7cb00a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 0x7f7cb00a24e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006010 tx=0x7f7cb8000bc0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 msgr2=0x7f7cc8068610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00d980 tx=0x7f7cbc00dd40 comp rx=0 tx=0).stop 2023-08-25T02:25:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 shutdown_connections 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cc8101390 0x7f7cc819d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cc80ff800 0x7f7cc8068610 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7cb00a0020 0x7f7cb00a24e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 --2- 172.21.15.89:0/1574101771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc81009f0 0x7f7cc819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 >> 172.21.15.89:0/1574101771 conn(0x7f7cc80fad50 msgr2=0x7f7cc8103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 shutdown_connections 2023-08-25T02:25:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.197+0000 7f7ccdae5700 1 -- 172.21.15.89:0/1574101771 wait complete. 2023-08-25T02:25:55.199 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:25:55.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:25:55.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 -- 172.21.15.89:0/3918723113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96301013b0 msgr2=0x7f9630107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 --2- 172.21.15.89:0/3918723113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96301013b0 0x7f9630107c60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f962800b0d0 tx=0x7f962800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 -- 172.21.15.89:0/3918723113 shutdown_connections 2023-08-25T02:25:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 --2- 172.21.15.89:0/3918723113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96301013b0 0x7f9630107c60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 --2- 172.21.15.89:0/3918723113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96300ff820 0x7f96300ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 --2- 172.21.15.89:0/3918723113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 -- 172.21.15.89:0/3918723113 >> 172.21.15.89:0/3918723113 conn(0x7f96300fad70 msgr2=0x7f96300fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 -- 172.21.15.89:0/3918723113 shutdown_connections 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 -- 172.21.15.89:0/3918723113 wait complete. 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.330+0000 7f96356e5700 1 Processor -- start 2023-08-25T02:25:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 -- start start 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 0x7f963019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963019e170 con 0x7f96300ff820 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963019e2b0 con 0x7f9630100a10 2023-08-25T02:25:55.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f96356e5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963019e440 con 0x7f96301013b0 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47376/0 (socket says 172.21.15.89:47376) 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 -- 172.21.15.89:0/171433743 learned_addr learned my addr 172.21.15.89:0/171433743 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962e7fc700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962f7fe700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 0x7f963019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 -- 172.21.15.89:0/171433743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 msgr2=0x7f963019dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 0x7f963019dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 -- 172.21.15.89:0/171433743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 msgr2=0x7f9630196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 -- 172.21.15.89:0/171433743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628009d00 con 0x7f96300ff820 2023-08-25T02:25:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962e7fc700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:55.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962f7fe700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 0x7f963019dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.331+0000 7f962effd700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f962000d980 tx=0x7f962000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.332+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000de60 con 0x7f96300ff820 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.332+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96200044e0 con 0x7f96300ff820 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.332+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620003be0 con 0x7f96300ff820 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.332+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963019e720 con 0x7f96300ff820 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.332+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963019ebe0 con 0x7f96300ff820 2023-08-25T02:25:55.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.333+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963010c1e0 con 0x7f96300ff820 2023-08-25T02:25:55.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.334+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f9620009bb0 con 0x7f96300ff820 2023-08-25T02:25:55.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.334+0000 7f9617fff700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 0x7f96180a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.334+0000 7f962e7fc700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 0x7f96180a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.334+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f96200caec0 con 0x7f96300ff820 2023-08-25T02:25:55.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.335+0000 7f962e7fc700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 0x7f96180a2460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9624005fd0 tx=0x7f9624005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:55.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.339+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f9620094da0 con 0x7f96300ff820 2023-08-25T02:25:55.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.530+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f963010c3d0 con 0x7f96300ff820 2023-08-25T02:25:55.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.532+0000 7f9617fff700 1 -- 172.21.15.89:0/171433743 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22107 (secure 0 0 0) 0x7f9620009940 con 0x7f96300ff820 2023-08-25T02:25:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:55.534 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":80,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:25:05.210667+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:25:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 msgr2=0x7f96180a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 0x7f96180a2460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9624005fd0 tx=0x7f9624005e30 comp rx=0 tx=0).stop 2023-08-25T02:25:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 msgr2=0x7f96301965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f962000d980 tx=0x7f962000dd40 comp rx=0 tx=0).stop 2023-08-25T02:25:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 shutdown_connections 2023-08-25T02:25:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96301013b0 0x7f963019dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300ff820 0x7f96301965f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f961809ffa0 0x7f96180a2460 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 --2- 172.21.15.89:0/171433743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9630100a10 0x7f9630196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 >> 172.21.15.89:0/171433743 conn(0x7f96300fad70 msgr2=0x7f96300fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:55.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 shutdown_connections 2023-08-25T02:25:55.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.535+0000 7f96356e5700 1 -- 172.21.15.89:0/171433743 wait complete. 2023-08-25T02:25:55.547 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:55 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2189057773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.547 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:55 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/965134812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.547 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:55 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1574101771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:55.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:25:55.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 -- 172.21.15.89:0/924371776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13140ffab0 msgr2=0x7f13140fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 --2- 172.21.15.89:0/924371776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13140ffab0 0x7f13140fff30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1304009990 tx=0x7f1304009ca0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 -- 172.21.15.89:0/924371776 shutdown_connections 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 --2- 172.21.15.89:0/924371776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 0x7f1314107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 --2- 172.21.15.89:0/924371776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1314100e60 0x7f1314101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 --2- 172.21.15.89:0/924371776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13140ffab0 0x7f13140fff30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 -- 172.21.15.89:0/924371776 >> 172.21.15.89:0/924371776 conn(0x7f13140fa520 msgr2=0x7f13140fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 -- 172.21.15.89:0/924371776 shutdown_connections 2023-08-25T02:25:55.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 -- 172.21.15.89:0/924371776 wait complete. 2023-08-25T02:25:55.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.669+0000 7f131930a700 1 Processor -- start 2023-08-25T02:25:55.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 -- start start 2023-08-25T02:25:55.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13140ffab0 0x7f13141954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 0x7f1314195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131419d040 con 0x7f1314100e60 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131419d180 con 0x7f1314100470 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f131930a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131419d310 con 0x7f13140ffab0 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47394/0 (socket says 172.21.15.89:47394) 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 -- 172.21.15.89:0/1834914130 learned_addr learned my addr 172.21.15.89:0/1834914130 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f1312ffd700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13140ffab0 0x7f13141954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13127fc700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 0x7f1314195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 -- 172.21.15.89:0/1834914130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13140ffab0 msgr2=0x7f13141954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13140ffab0 0x7f13141954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 -- 172.21.15.89:0/1834914130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 msgr2=0x7f1314195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 0x7f1314195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 -- 172.21.15.89:0/1834914130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040095f0 con 0x7f1314100e60 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f13137fe700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f130c007ee0 tx=0x7f130c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.670+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c017070 con 0x7f1314100e60 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.671+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131419d5f0 con 0x7f1314100e60 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.671+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f130c00f040 con 0x7f1314100e60 2023-08-25T02:25:55.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.671+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c005550 con 0x7f1314100e60 2023-08-25T02:25:55.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.671+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131419db50 con 0x7f1314100e60 2023-08-25T02:25:55.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.671+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1314106e40 con 0x7f1314100e60 2023-08-25T02:25:55.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.673+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f130c00db20 con 0x7f1314100e60 2023-08-25T02:25:55.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.673+0000 7f130bfff700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 0x7f13000a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:55.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.673+0000 7f1312ffd700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 0x7f13000a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:55.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.673+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f130c0cae80 con 0x7f1314100e60 2023-08-25T02:25:55.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.674+0000 7f1312ffd700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 0x7f13000a2460 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1314068ac0 tx=0x7f12fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:55.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.677+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f130c094d60 con 0x7f1314100e60 2023-08-25T02:25:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:55 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2189057773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:55 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/965134812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:55 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1574101771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:55 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2189057773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:55 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/965134812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:25:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:55 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1574101771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.917+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f131410b0a0 con 0x7f1314100e60 2023-08-25T02:25:55.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.919+0000 7f130bfff700 1 -- 172.21.15.89:0/1834914130 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7f130c098380 con 0x7f1314100e60 2023-08-25T02:25:55.920 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:25:55.920 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:25:55.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.921+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 msgr2=0x7f13000a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.921+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 0x7f13000a2460 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1314068ac0 tx=0x7f12fc009260 comp rx=0 tx=0).stop 2023-08-25T02:25:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.921+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 msgr2=0x7f131419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.921+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f130c007ee0 tx=0x7f130c00d3b0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 shutdown_connections 2023-08-25T02:25:55.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1314100e60 0x7f131419ca70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13140ffab0 0x7f13141954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f130009ffa0 0x7f13000a2460 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 --2- 172.21.15.89:0/1834914130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314100470 0x7f1314195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 >> 172.21.15.89:0/1834914130 conn(0x7f13140fa520 msgr2=0x7f1314103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 shutdown_connections 2023-08-25T02:25:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:55.922+0000 7f131930a700 1 -- 172.21.15.89:0/1834914130 wait complete. 2023-08-25T02:25:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:25:55.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-08-25T02:25:04.024043+0000', 'modified': '2023-08-25T02:25:09.635801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24475, 'mds_2': 24470}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.121:6835/217468698', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 217468698}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 217468698}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6835/3100558972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 3100558972}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 3100558972}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6837/2359799361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2359799361}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2359799361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.121:6837/2048790446', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6836', 'nonce': 2048790446}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 2048790446}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.122:6839/3754052688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 3754052688}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 3754052688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6839/1864669733', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6838', 'nonce': 1864669733}, {'type': 'v1', 'addr': '172.21.15.121:6839', 'nonce': 1864669733}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-08-25T02:25:55.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-08-25T02:25:55.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-08-25T02:25:56.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- 172.21.15.89:0/249860862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e180fef80 msgr2=0x7f0e180ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/249860862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e180fef80 0x7f0e180ff3a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0099c0 tx=0x7f0e0c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- 172.21.15.89:0/249860862 shutdown_connections 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/249860862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100b10 0x7f0e181073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/249860862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e180fef80 0x7f0e180ff3a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/249860862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e18100170 0x7f0e181005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- 172.21.15.89:0/249860862 >> 172.21.15.89:0/249860862 conn(0x7f0e180fa4f0 msgr2=0x7f0e180fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- 172.21.15.89:0/249860862 shutdown_connections 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- 172.21.15.89:0/249860862 wait complete. 2023-08-25T02:25:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 Processor -- start 2023-08-25T02:25:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.059+0000 7f0e1d95b700 1 -- start start 2023-08-25T02:25:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e180fef80 0x7f0e181988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100170 0x7f0e18198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e181a0420 con 0x7f0e18100b10 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e181a0560 con 0x7f0e18100170 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e181a06f0 con 0x7f0e180fef80 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47408/0 (socket says 172.21.15.89:47408) 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 -- 172.21.15.89:0/671238003 learned_addr learned my addr 172.21.15.89:0/671238003 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e167fc700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100170 0x7f0e18198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e180fef80 0x7f0e181988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 -- 172.21.15.89:0/671238003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e180fef80 msgr2=0x7f0e181988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e180fef80 0x7f0e181988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 -- 172.21.15.89:0/671238003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100170 msgr2=0x7f0e18198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100170 0x7f0e18198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 -- 172.21.15.89:0/671238003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0095f0 con 0x7f0e18100b10 2023-08-25T02:25:56.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e177fe700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000ada0 tx=0x7f0e1000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1c959700 1 -- 172.21.15.89:0/671238003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1000f070 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1c959700 1 -- 172.21.15.89:0/671238003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e10007b30 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e181a09d0 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.060+0000 7f0e1c959700 1 -- 172.21.15.89:0/671238003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100053e0 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.061+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e181a0ee0 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.063+0000 7f0e1c959700 1 -- 172.21.15.89:0/671238003 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0e1000c980 con 0x7f0e18100b10 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.063+0000 7f0e1c959700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 0x7f0e000a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.063+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 0x7f0e000a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.063+0000 7f0e1c959700 1 -- 172.21.15.89:0/671238003 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0e100ca500 con 0x7f0e18100b10 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 0x7f0e000a2410 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00b580 tx=0x7f0e0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 Processor -- start 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 -- start start 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc0503d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0503e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04fa4760 con 0x55cc0503c980 2023-08-25T02:25:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f95730 con 0x55cc0503dd00 2023-08-25T02:25:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e29affb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f93050 con 0x55cc0503d2e0 2023-08-25T02:25:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47412/0 (socket says 172.21.15.89:47412) 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 -- 172.21.15.89:0/4187064654 learned_addr learned my addr 172.21.15.89:0/4187064654 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 -- 172.21.15.89:0/4187064654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 msgr2=0x55cc0503d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc0503d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 -- 172.21.15.89:0/4187064654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 msgr2=0x55cc0503e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0503e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 -- 172.21.15.89:0/4187064654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc04e0e5e0 con 0x55cc0503c980 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.064+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0e180fffd0 tx=0x7f0e0c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987fb465ee3cbf74 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0dfdffb700 1 -- 172.21.15.89:0/4187064654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c02b070 con 0x55cc0503c980 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0dfdffb700 1 -- 172.21.15.89:0/4187064654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e0c01e040 con 0x55cc0503c980 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0dfdffb700 1 -- 172.21.15.89:0/4187064654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c004060 con 0x55cc0503c980 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 -- 172.21.15.89:0/4187064654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 msgr2=0x55cc0503cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0e180fffd0 tx=0x7f0e0c01f000 comp rx=0 tx=0).stop 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 -- 172.21.15.89:0/4187064654 shutdown_connections 2023-08-25T02:25:56.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0503e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc0503cda0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 --2- 172.21.15.89:0/4187064654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc0503d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 -- 172.21.15.89:0/4187064654 >> 172.21.15.89:0/4187064654 conn(0x55cc04fb1090 msgr2=0x55cc05037250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 -- 172.21.15.89:0/4187064654 shutdown_connections 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.065+0000 7f0e29affb80 1 -- 172.21.15.89:0/4187064654 wait complete. 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 -- start start 2023-08-25T02:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 Processor -- start 2023-08-25T02:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc050468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0504d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f8e940 con 0x55cc0503c980 2023-08-25T02:25:56.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f2cb60 con 0x55cc0503dd00 2023-08-25T02:25:56.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f2c860 con 0x55cc0503d2e0 2023-08-25T02:25:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e29affb80 10 client.0 _ll_register_callbacks cb 0x55cc04e76a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47418/0 (socket says 172.21.15.89:47418) 2023-08-25T02:25:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 -- 172.21.15.89:0/2543723007 learned_addr learned my addr 172.21.15.89:0/2543723007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 -- 172.21.15.89:0/2543723007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 msgr2=0x55cc050468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc050468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 -- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 msgr2=0x55cc0504d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0504d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.066+0000 7f0e16ffd700 1 -- 172.21.15.89:0/2543723007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e18107d10 con 0x55cc0503c980 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c031040 tx=0x7f0e0c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ff7da4d9267b62 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c02b070 con 0x55cc0503c980 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0e0c00bbe0 con 0x55cc0503c980 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc0500c170 con 0x55cc0503c980 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e0c00bda0 con 0x55cc0503c980 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 10 client.15009 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cc04dbad90 con 0x55cc0503c980 2023-08-25T02:25:56.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc0500c170 con 0x55cc0503c980 2023-08-25T02:25:56.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e0c00f560 con 0x55cc0503c980 2023-08-25T02:25:56.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 10 client.15009 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 10 client.15009 resolve_mds: resolved name 'h' to daemon mds.h 2023-08-25T02:25:56.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 20 client.15009 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc05050840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 4 client.15009 mds_command: new command op to 24470 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:25:56.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.067+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 --> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc04f34b50 con 0x55cc05051b40 2023-08-25T02:25:56.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.068+0000 7f0e167fc700 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc05050840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.068+0000 7f0e167fc700 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc05050840 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f73fbe9f16fc46b6 server_cookie=ec4d24e1482f908f in_seq=0 out_seq=0 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.068+0000 7f0df6ffd700 10 client.15009 ms_handle_connect on v2:172.21.15.121:6836/2048790446 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0df6ffd700 1 -- 172.21.15.89:0/2543723007 <== mds.2 v2:172.21.15.121:6836/2048790446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cc04f34b50 con 0x55cc05051b40 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0df6ffd700 10 client.15009 handle_command_reply: tid=0 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 client.15009 shutdown 2023-08-25T02:25:56.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 msgr2=0x55cc05046390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c031040 tx=0x7f0e0c005c00 comp rx=0 tx=0).stop 2023-08-25T02:25:56.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 shutdown_connections 2023-08-25T02:25:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc05050840 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503dd00 0x55cc0504d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0df6ffd700 0 client.15009 ms_handle_reset on v2:172.21.15.121:6836/2048790446 2023-08-25T02:25:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503c980 0x55cc05046390 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 --2- 172.21.15.89:0/2543723007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503d2e0 0x55cc050468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 >> 172.21.15.89:0/2543723007 conn(0x55cc04fb1090 msgr2=0x55cc05031af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 shutdown_connections 2023-08-25T02:25:56.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 1 -- 172.21.15.89:0/2543723007 wait complete. 2023-08-25T02:25:56.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.069+0000 7f0e29affb80 20 client.15009 trim_cache size 0 max 16384 2023-08-25T02:25:56.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.085+0000 7f0e29affb80 1 Processor -- start 2023-08-25T02:25:56.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 -- start start 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc05045030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04e108d0 con 0x55cc0503dd00 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc04f34b80 con 0x55cc0503d2e0 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e29affb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc050473f0 con 0x55cc0503c980 2023-08-25T02:25:56.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47430/0 (socket says 172.21.15.89:47430) 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 -- 172.21.15.89:0/1504697242 learned_addr learned my addr 172.21.15.89:0/1504697242 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc05045030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 -- 172.21.15.89:0/1504697242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 msgr2=0x55cc05045030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc05045030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 -- 172.21.15.89:0/1504697242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 msgr2=0x55cc0504d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e167fc700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 -- 172.21.15.89:0/1504697242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc04da3c10 con 0x55cc0503dd00 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e167fc700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0e177fe700 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0e1009f980 tx=0x7f0e10097c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac24b9760d87bbe0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0dfe7fc700 1 -- 172.21.15.89:0/1504697242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100b0210 con 0x55cc0503dd00 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0dfe7fc700 1 -- 172.21.15.89:0/1504697242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e1009a030 con 0x55cc0503dd00 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.086+0000 7f0dfe7fc700 1 -- 172.21.15.89:0/1504697242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1000a1d0 con 0x55cc0503dd00 2023-08-25T02:25:56.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- 172.21.15.89:0/1504697242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 msgr2=0x55cc05040ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0e1009f980 tx=0x7f0e10097c70 comp rx=0 tx=0).stop 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- 172.21.15.89:0/1504697242 shutdown_connections 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc05040ca0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc05045030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 --2- 172.21.15.89:0/1504697242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- 172.21.15.89:0/1504697242 >> 172.21.15.89:0/1504697242 conn(0x55cc04fb1090 msgr2=0x55cc05044160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- 172.21.15.89:0/1504697242 shutdown_connections 2023-08-25T02:25:56.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- 172.21.15.89:0/1504697242 wait complete. 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 -- start start 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.087+0000 7f0e29affb80 1 Processor -- start 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc0504d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0504acb0 con 0x55cc0503dd00 2023-08-25T02:25:56.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc0504ae20 con 0x55cc0503d2e0 2023-08-25T02:25:56.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc050a5530 con 0x55cc0503c980 2023-08-25T02:25:56.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 10 client.0 _ll_register_callbacks cb 0x7f0e0c0053c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47436/0 (socket says 172.21.15.89:47436) 2023-08-25T02:25:56.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 -- 172.21.15.89:0/3846861906 learned_addr learned my addr 172.21.15.89:0/3846861906 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e167fc700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc0504d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 -- 172.21.15.89:0/3846861906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 msgr2=0x55cc0504d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc0504d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 -- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 msgr2=0x55cc0504a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 -- 172.21.15.89:0/3846861906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e181a0ee0 con 0x55cc0503dd00 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc0504d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e177fe700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e10097e30 tx=0x7f0e10004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9300bd05df28b0a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0df6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10098830 con 0x55cc0503dd00 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0e1000a1d0 con 0x55cc0503dd00 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.088+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc0502f860 con 0x55cc0503dd00 2023-08-25T02:25:56.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e10098390 con 0x55cc0503dd00 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 10 client.15015 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cc04de1200 con 0x55cc0503dd00 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc0502f860 con 0x55cc0503dd00 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e100ca0d0 con 0x55cc0503dd00 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 10 client.15015 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 10 client.15015 resolve_mds: resolved name 'h' to daemon mds.h 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 20 client.15015 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc050a6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 4 client.15015 mds_command: new command op to 24470 tid=0[{"prefix": "status"}] 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 --> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cc04f87920 con 0x55cc05051b40 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc050a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0df6ffd700 10 client.15015 ms_handle_connect on v2:172.21.15.121:6836/2048790446 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.089+0000 7f0e16ffd700 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc050a6800 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e874241c201084ab server_cookie=65819670a772be62 in_seq=0 out_seq=0 2023-08-25T02:25:56.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0df6ffd700 1 -- 172.21.15.89:0/3846861906 <== mds.2 v2:172.21.15.121:6836/2048790446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f0e0c01e160 con 0x55cc05051b40 2023-08-25T02:25:56.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0df6ffd700 10 client.15015 handle_command_reply: tid=0 2023-08-25T02:25:56.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 client.15015 shutdown 2023-08-25T02:25:56.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 msgr2=0x55cc0504a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e10097e30 tx=0x7f0e10004c00 comp rx=0 tx=0).stop 2023-08-25T02:25:56.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 shutdown_connections 2023-08-25T02:25:56.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x55cc05051b40 0x55cc050a6800 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc0503dd00 0x55cc0504a770 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cc0503c980 0x55cc0504d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 --2- 172.21.15.89:0/3846861906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc0503d2e0 0x55cc0504a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 >> 172.21.15.89:0/3846861906 conn(0x55cc04fb1090 msgr2=0x55cc05045030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0df6ffd700 0 client.15015 ms_handle_reset on v2:172.21.15.121:6836/2048790446 2023-08-25T02:25:56.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 shutdown_connections 2023-08-25T02:25:56.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 1 -- 172.21.15.89:0/3846861906 wait complete. 2023-08-25T02:25:56.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.090+0000 7f0e29affb80 20 client.15015 trim_cache size 0 max 16384 2023-08-25T02:25:56.108 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:25:56.108 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:25:56.108 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2023-08-25T02:25:56.108 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24470, 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 47.459758438999998, 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 80, 2023-08-25T02:25:56.109 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:25:56.110 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 65.543596257999994 2023-08-25T02:25:56.110 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:25:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 msgr2=0x7f0e000a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 0x7f0e000a2410 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00b580 tx=0x7f0e0c005fb0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 msgr2=0x7f0e1819fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000ada0 tx=0x7f0e1000c330 comp rx=0 tx=0).stop 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 shutdown_connections 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e18100b10 0x7f0e1819fe50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e180fef80 0x7f0e181988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0e0009ff50 0x7f0e000a2410 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 --2- 172.21.15.89:0/671238003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e18100170 0x7f0e18198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 >> 172.21.15.89:0/671238003 conn(0x7f0e180fa4f0 msgr2=0x7f0e180fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 shutdown_connections 2023-08-25T02:25:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.091+0000 7f0e1d95b700 1 -- 172.21.15.89:0/671238003 wait complete. 2023-08-25T02:25:56.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-08-25T02:25:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 -- 172.21.15.89:0/2976581576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc0ff810 msgr2=0x7f7cfc0ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2976581576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc0ffc30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0099c0 tx=0x7f7cec009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 -- 172.21.15.89:0/2976581576 shutdown_connections 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2976581576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 0x7f7cfc107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2976581576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc0ffc30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2976581576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100a00 0x7f7cfc100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 -- 172.21.15.89:0/2976581576 >> 172.21.15.89:0/2976581576 conn(0x7f7cfc0fad80 msgr2=0x7f7cfc0fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 -- 172.21.15.89:0/2976581576 shutdown_connections 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.233+0000 7f7d0336e700 1 -- 172.21.15.89:0/2976581576 wait complete. 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 Processor -- start 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 -- start start 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 0x7f7cfc19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1a0370 con 0x7f7cfc100a00 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1a04b0 con 0x7f7cfc0ff810 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0336e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1a0640 con 0x7f7cfc1013a0 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47446/0 (socket says 172.21.15.89:47446) 2023-08-25T02:25:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 -- 172.21.15.89:0/2613974150 learned_addr learned my addr 172.21.15.89:0/2613974150 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0110a700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d0190b700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 0x7f7cfc19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 -- 172.21.15.89:0/2613974150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 msgr2=0x7f7cfc19fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 0x7f7cfc19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 -- 172.21.15.89:0/2613974150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc0ff810 msgr2=0x7f7cfc1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.234+0000 7f7d00909700 1 -- 172.21.15.89:0/2613974150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec0095f0 con 0x7f7cfc100a00 2023-08-25T02:25:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7d00909700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800c9b0 tx=0x7f7cf800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7cf27fc700 1 -- 172.21.15.89:0/2613974150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8007b00 con 0x7f7cfc100a00 2023-08-25T02:25:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7cf27fc700 1 -- 172.21.15.89:0/2613974150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf8007c60 con 0x7f7cfc100a00 2023-08-25T02:25:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc1a0920 con 0x7f7cfc100a00 2023-08-25T02:25:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7cf27fc700 1 -- 172.21.15.89:0/2613974150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8018660 con 0x7f7cfc100a00 2023-08-25T02:25:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.235+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc1a0e30 con 0x7f7cfc100a00 2023-08-25T02:25:56.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.237+0000 7f7cf27fc700 1 -- 172.21.15.89:0/2613974150 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f7cf800f450 con 0x7f7cfc100a00 2023-08-25T02:25:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.237+0000 7f7cf27fc700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 0x7f7ce80a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.238+0000 7f7d0110a700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 0x7f7ce80a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.238+0000 7f7cf27fc700 1 -- 172.21.15.89:0/2613974150 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7cf80cb0f0 con 0x7f7cfc100a00 2023-08-25T02:25:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.238+0000 7f7d0110a700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 0x7f7ce80a2410 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00b580 tx=0x7f7cec009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 Processor -- start 2023-08-25T02:25:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 -- start start 2023-08-25T02:25:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x56539649cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x56539649e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565396404760 con 0x56539649d2e0 2023-08-25T02:25:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653963f5730 con 0x56539649c980 2023-08-25T02:25:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653963f3050 con 0x56539649dd00 2023-08-25T02:25:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47454/0 (socket says 172.21.15.89:47454) 2023-08-25T02:25:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 -- 172.21.15.89:0/4009586291 learned_addr learned my addr 172.21.15.89:0/4009586291 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0110a700 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x56539649cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 -- 172.21.15.89:0/4009586291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 msgr2=0x56539649e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x56539649e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 -- 172.21.15.89:0/4009586291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 msgr2=0x56539649cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x56539649cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 -- 172.21.15.89:0/4009586291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56539626e5e0 con 0x56539649d2e0 2023-08-25T02:25:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d0110a700 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x56539649cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7d00909700 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c8d60 tx=0x7f7cf8098900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ee330568f7fa1d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7ce77fe700 1 -- 172.21.15.89:0/4009586291 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf809cac0 con 0x56539649d2e0 2023-08-25T02:25:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7ce77fe700 1 -- 172.21.15.89:0/4009586291 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf80a0aa0 con 0x56539649d2e0 2023-08-25T02:25:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.239+0000 7f7ce77fe700 1 -- 172.21.15.89:0/4009586291 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800f960 con 0x56539649d2e0 2023-08-25T02:25:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4009586291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 msgr2=0x56539649d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c8d60 tx=0x7f7cf8098900 comp rx=0 tx=0).stop 2023-08-25T02:25:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4009586291 shutdown_connections 2023-08-25T02:25:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x56539649cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x56539649e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4009586291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x56539649d720 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4009586291 >> 172.21.15.89:0/4009586291 conn(0x565396411090 msgr2=0x565396497250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4009586291 shutdown_connections 2023-08-25T02:25:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4009586291 wait complete. 2023-08-25T02:25:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.240+0000 7f7d0f512b80 1 -- start start 2023-08-25T02:25:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 Processor -- start 2023-08-25T02:25:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649c980 0x5653964a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649dd00 0x5653964ad920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653963ee940 con 0x56539649d2e0 2023-08-25T02:25:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56539638cb60 con 0x56539649dd00 2023-08-25T02:25:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56539638c860 con 0x56539649c980 2023-08-25T02:25:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0f512b80 10 client.0 _ll_register_callbacks cb 0x5653962d6a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47460/0 (socket says 172.21.15.89:47460) 2023-08-25T02:25:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 -- 172.21.15.89:0/4274793837 learned_addr learned my addr 172.21.15.89:0/4274793837 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0110a700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649c980 0x5653964a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d0190b700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649dd00 0x5653964ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 -- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649c980 msgr2=0x5653964a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649c980 0x5653964a63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 -- 172.21.15.89:0/4274793837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649dd00 msgr2=0x5653964ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649dd00 0x5653964ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.241+0000 7f7d00909700 1 -- 172.21.15.89:0/4274793837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc1a0e30 con 0x56539649d2e0 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d00909700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c8d90 tx=0x7f7cf8097230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a4e0ed3ec557be server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf80990f0 con 0x56539649d2e0 2023-08-25T02:25:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7cf8060050 con 0x56539649d2e0 2023-08-25T02:25:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56539646c170 con 0x56539649d2e0 2023-08-25T02:25:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cf809cac0 con 0x56539649d2e0 2023-08-25T02:25:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 10 client.15027 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56539621ad90 con 0x56539649d2e0 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56539646c170 con 0x56539649d2e0 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7cf8062070 con 0x56539649d2e0 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 10 client.15027 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 10 client.15027 resolve_mds: resolved name 'i' to daemon mds.i 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 20 client.15027 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x5653964b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 4 client.15027 mds_command: new command op to 24475 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.242+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 --> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565396394b50 con 0x5653964b1980 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.243+0000 7f7d0110a700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x5653964b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.243+0000 7f7ce4ff9700 10 client.15027 ms_handle_connect on v2:172.21.15.122:6838/3754052688 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.243+0000 7f7d0110a700 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x5653964b07c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ccdc1ef65b7c7961 server_cookie=c64abebf913a7dbe in_seq=0 out_seq=0 2023-08-25T02:25:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/4274793837 <== mds.1 v2:172.21.15.122:6838/3754052688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x565396394b50 con 0x5653964b1980 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7ce4ff9700 10 client.15027 handle_command_reply: tid=0 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 client.15027 shutdown 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 msgr2=0x5653964a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80c8d90 tx=0x7f7cf8097230 comp rx=0 tx=0).stop 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 shutdown_connections 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x5653964b07c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649dd00 0x5653964ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7ce4ff9700 0 client.15027 ms_handle_reset on v2:172.21.15.122:6838/3754052688 2023-08-25T02:25:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649c980 0x5653964a63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/4274793837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964a68e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 >> 172.21.15.89:0/4274793837 conn(0x565396411090 msgr2=0x565396491af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 shutdown_connections 2023-08-25T02:25:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 1 -- 172.21.15.89:0/4274793837 wait complete. 2023-08-25T02:25:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.244+0000 7f7d0f512b80 20 client.15027 trim_cache size 0 max 16384 2023-08-25T02:25:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 Processor -- start 2023-08-25T02:25:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 -- start start 2023-08-25T02:25:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x5653964a5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653962708d0 con 0x56539649d2e0 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x565396498700 con 0x56539649c980 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x565396396470 con 0x56539649dd00 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47470/0 (socket says 172.21.15.89:47470) 2023-08-25T02:25:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 -- 172.21.15.89:0/959399279 learned_addr learned my addr 172.21.15.89:0/959399279 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0110a700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x5653964a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0190b700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 -- 172.21.15.89:0/959399279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 msgr2=0x5653964b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 -- 172.21.15.89:0/959399279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 msgr2=0x5653964a5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x5653964a5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 -- 172.21.15.89:0/959399279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565396203c10 con 0x56539649d2e0 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d0190b700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7d00909700 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8098990 tx=0x7f7cf8094fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e059a220b92d83 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.262+0000 7f7ce7fff700 1 -- 172.21.15.89:0/959399279 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8098380 con 0x56539649d2e0 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7ce7fff700 1 -- 172.21.15.89:0/959399279 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf8077e90 con 0x56539649d2e0 2023-08-25T02:25:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7ce7fff700 1 -- 172.21.15.89:0/959399279 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8098ef0 con 0x56539649d2e0 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- 172.21.15.89:0/959399279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 msgr2=0x5653964ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8098990 tx=0x7f7cf8094fd0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- 172.21.15.89:0/959399279 shutdown_connections 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649c980 0x5653964a5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/959399279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649d2e0 0x5653964ad8e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- 172.21.15.89:0/959399279 >> 172.21.15.89:0/959399279 conn(0x565396411090 msgr2=0x565396490ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- 172.21.15.89:0/959399279 shutdown_connections 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- 172.21.15.89:0/959399279 wait complete. 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 -- start start 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.263+0000 7f7d0f512b80 1 Processor -- start 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 0x5653964aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964acf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653964ad4c0 con 0x56539649c980 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653964ad630 con 0x56539649d2e0 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653964ad7f0 con 0x56539649dd00 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 10 client.0 _ll_register_callbacks cb 0x7f7cf80987d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47472/0 (socket says 172.21.15.89:47472) 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 -- 172.21.15.89:0/3841516767 learned_addr learned my addr 172.21.15.89:0/3841516767 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d00909700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 0x5653964aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 -- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 msgr2=0x5653964acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 -- 172.21.15.89:0/3841516767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 msgr2=0x5653964aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 0x5653964aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 -- 172.21.15.89:0/3841516767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56539621ad90 con 0x56539649c980 2023-08-25T02:25:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d00909700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 0x5653964aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0110a700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006010 tx=0x7f7cec009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbe2f6de142d52f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7ce4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec033ad0 con 0x56539649c980 2023-08-25T02:25:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7cec033c30 con 0x56539649c980 2023-08-25T02:25:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.264+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56539648f860 con 0x56539649c980 2023-08-25T02:25:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cec033df0 con 0x56539649c980 2023-08-25T02:25:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 10 client.15033 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x565396241200 con 0x56539649c980 2023-08-25T02:25:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56539648f860 con 0x56539649c980 2023-08-25T02:25:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7cec046070 con 0x56539649c980 2023-08-25T02:25:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 10 client.15033 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 10 client.15033 resolve_mds: resolved name 'i' to daemon mds.i 2023-08-25T02:25:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 20 client.15033 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x565396506cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 4 client.15033 mds_command: new command op to 24475 tid=0[{"prefix": "status"}] 2023-08-25T02:25:56.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 --> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653963e7920 con 0x5653964b1980 2023-08-25T02:25:56.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.265+0000 7f7d00909700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x565396506cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7ce4ff9700 10 client.15033 ms_handle_connect on v2:172.21.15.122:6838/3754052688 2023-08-25T02:25:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d00909700 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x565396506cc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc85a9efa49c6211 server_cookie=3f949185e76a6740 in_seq=0 out_seq=0 2023-08-25T02:25:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7ce4ff9700 1 -- 172.21.15.89:0/3841516767 <== mds.1 v2:172.21.15.122:6838/3754052688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5653963e7920 con 0x5653964b1980 2023-08-25T02:25:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7ce4ff9700 10 client.15033 handle_command_reply: tid=0 2023-08-25T02:25:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 client.15033 shutdown 2023-08-25T02:25:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 msgr2=0x5653964a9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006010 tx=0x7f7cec009310 comp rx=0 tx=0).stop 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 shutdown_connections 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x5653964b1980 0x565396506cc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56539649dd00 0x5653964acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56539649c980 0x5653964a9d30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 --2- 172.21.15.89:0/3841516767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56539649d2e0 0x5653964aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7ce4ff9700 0 client.15033 ms_handle_reset on v2:172.21.15.122:6838/3754052688 2023-08-25T02:25:56.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 >> 172.21.15.89:0/3841516767 conn(0x565396411090 msgr2=0x5653964a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 shutdown_connections 2023-08-25T02:25:56.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 1 -- 172.21.15.89:0/3841516767 wait complete. 2023-08-25T02:25:56.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.266+0000 7f7d0f512b80 20 client.15033 trim_cache size 0 max 16384 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24475, 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:25:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 48.644223330000003, 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 80, 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 63.559661101000003 2023-08-25T02:25:56.298 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:25:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 msgr2=0x7f7ce80a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 0x7f7ce80a2410 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00b580 tx=0x7f7cec009f10 comp rx=0 tx=0).stop 2023-08-25T02:25:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 msgr2=0x7f7cfc198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800c9b0 tx=0x7f7cf800ccc0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 shutdown_connections 2023-08-25T02:25:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f7ce809ff50 0x7f7ce80a2410 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc0ff810 0x7f7cfc1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cfc1013a0 0x7f7cfc19fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 --2- 172.21.15.89:0/2613974150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cfc100a00 0x7f7cfc198d30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 >> 172.21.15.89:0/2613974150 conn(0x7f7cfc0fad80 msgr2=0x7f7cfc067b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 shutdown_connections 2023-08-25T02:25:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.267+0000 7f7d0336e700 1 -- 172.21.15.89:0/2613974150 wait complete. 2023-08-25T02:25:56.303 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-08-25T02:25:56.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 -- 172.21.15.89:0/3241203417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 msgr2=0x7fe7780ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 --2- 172.21.15.89:0/3241203417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe7780ff3a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0099c0 tx=0x7fe76c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 -- 172.21.15.89:0/3241203417 shutdown_connections 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 --2- 172.21.15.89:0/3241203417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100b10 0x7fe7781073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 --2- 172.21.15.89:0/3241203417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe7780ff3a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 --2- 172.21.15.89:0/3241203417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe778100170 0x7fe7781005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 -- 172.21.15.89:0/3241203417 >> 172.21.15.89:0/3241203417 conn(0x7fe7780fa4f0 msgr2=0x7fe7780fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 -- 172.21.15.89:0/3241203417 shutdown_connections 2023-08-25T02:25:56.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.426+0000 7fe77dde3700 1 -- 172.21.15.89:0/3241203417 wait complete. 2023-08-25T02:25:56.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 Processor -- start 2023-08-25T02:25:56.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 -- start start 2023-08-25T02:25:56.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 0x7fe778196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe778100b10 0x7fe77819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77819e1e0 con 0x7fe7780fef80 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77819e320 con 0x7fe778100170 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe77dde3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77819e4b0 con 0x7fe778100b10 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47486/0 (socket says 172.21.15.89:47486) 2023-08-25T02:25:56.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 -- 172.21.15.89:0/1387004752 learned_addr learned my addr 172.21.15.89:0/1387004752 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 0x7fe778196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 -- 172.21.15.89:0/1387004752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe778100b10 msgr2=0x7fe77819dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe778100b10 0x7fe77819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 -- 172.21.15.89:0/1387004752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 msgr2=0x7fe778196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 0x7fe778196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe7777fe700 1 -- 172.21.15.89:0/1387004752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c0095f0 con 0x7fe7780fef80 2023-08-25T02:25:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.427+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 0x7fe778196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009fd0 tx=0x7fe76c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe774ff9700 1 -- 172.21.15.89:0/1387004752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c01d070 con 0x7fe7780fef80 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe774ff9700 1 -- 172.21.15.89:0/1387004752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe76c0054f0 con 0x7fe7780fef80 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77819e730 con 0x7fe7780fef80 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe774ff9700 1 -- 172.21.15.89:0/1387004752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c004bd0 con 0x7fe7780fef80 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.428+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77819eb50 con 0x7fe7780fef80 2023-08-25T02:25:56.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.430+0000 7fe774ff9700 1 -- 172.21.15.89:0/1387004752 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fe76c00fbb0 con 0x7fe7780fef80 2023-08-25T02:25:56.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.430+0000 7fe774ff9700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 0x7fe7600a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.430+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 0x7fe7600a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.430+0000 7fe774ff9700 1 -- 172.21.15.89:0/1387004752 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe76c0cbee0 con 0x7fe7780fef80 2023-08-25T02:25:56.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.431+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 0x7fe7600a2410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe768005fd0 tx=0x7fe768005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.431+0000 7fe789f87b80 1 Processor -- start 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.431+0000 7fe789f87b80 1 -- start start 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a10da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a11720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5978760 con 0x5617b5a11d00 2023-08-25T02:25:56.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5969730 con 0x5617b5a10980 2023-08-25T02:25:56.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe789f87b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5967050 con 0x5617b5a112e0 2023-08-25T02:25:56.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47498/0 (socket says 172.21.15.89:47498) 2023-08-25T02:25:56.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 -- 172.21.15.89:0/1676055568 learned_addr learned my addr 172.21.15.89:0/1676055568 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a10da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a11720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 -- 172.21.15.89:0/1676055568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 msgr2=0x5617b5a11720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a11720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 -- 172.21.15.89:0/1676055568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 msgr2=0x5617b5a10da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a10da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 -- 172.21.15.89:0/1676055568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617b57e25e0 con 0x5617b5a11d00 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a11720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:56.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe777fff700 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e400 tx=0x7fe77000e710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a627279290941f78 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe75dffb700 1 -- 172.21.15.89:0/1676055568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770018070 con 0x5617b5a11d00 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe75dffb700 1 -- 172.21.15.89:0/1676055568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe77000f040 con 0x5617b5a11d00 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.432+0000 7fe75dffb700 1 -- 172.21.15.89:0/1676055568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770004080 con 0x5617b5a11d00 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- 172.21.15.89:0/1676055568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 msgr2=0x5617b5a12180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e400 tx=0x7fe77000e710 comp rx=0 tx=0).stop 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- 172.21.15.89:0/1676055568 shutdown_connections 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a10da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a12180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1676055568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a11720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- 172.21.15.89:0/1676055568 >> 172.21.15.89:0/1676055568 conn(0x5617b5985090 msgr2=0x5617b5a0b250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- 172.21.15.89:0/1676055568 shutdown_connections 2023-08-25T02:25:56.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- 172.21.15.89:0/1676055568 wait complete. 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.433+0000 7fe789f87b80 1 -- start start 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 Processor -- start 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a10980 0x5617b5a1a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a112e0 0x5617b5a1a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5962940 con 0x5617b5a11d00 2023-08-25T02:25:56.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5900b60 con 0x5617b5a112e0 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5900860 con 0x5617b5a10980 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 10 client.0 _ll_register_callbacks cb 0x5617b584aa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47504/0 (socket says 172.21.15.89:47504) 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 -- 172.21.15.89:0/1545789016 learned_addr learned my addr 172.21.15.89:0/1545789016 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a10980 0x5617b5a1a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a112e0 0x5617b5a1a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 -- 172.21.15.89:0/1545789016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a10980 msgr2=0x5617b5a1a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a10980 0x5617b5a1a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 -- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a112e0 msgr2=0x5617b5a1a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a112e0 0x5617b5a1a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 -- 172.21.15.89:0/1545789016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77000e140 con 0x5617b5a11d00 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe777fff700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe770009ba0 tx=0x7fe770004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e358d1042aaa952 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe756ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770018040 con 0x5617b5a11d00 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe770004220 con 0x5617b5a11d00 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.434+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617b59e0170 con 0x5617b5a11d00 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7700043e0 con 0x5617b5a11d00 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 10 client.15045 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5617b578ed90 con 0x5617b5a11d00 2023-08-25T02:25:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5617b59e0170 con 0x5617b5a11d00 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe770014480 con 0x5617b5a11d00 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 10 client.15045 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 10 client.15045 resolve_mds: resolved name 'b' to daemon mds.b 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 20 client.15045 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a25e10 0x5617b5a248d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 4 client.15045 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617b5908b50 con 0x5617b5a25e10 2023-08-25T02:25:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.435+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a25e10 0x5617b5a248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.436+0000 7fe756ffd700 10 client.15045 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.436+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a25e10 0x5617b5a248d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9302c532e9e55c9 server_cookie=c98f224781edde68 in_seq=0 out_seq=0 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.436+0000 7fe756ffd700 1 -- 172.21.15.89:0/1545789016 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5617b5908b50 con 0x5617b5a25e10 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.436+0000 7fe756ffd700 10 client.15045 handle_command_reply: tid=0 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.436+0000 7fe789f87b80 1 client.15045 shutdown 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 msgr2=0x5617b5a21880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe770009ba0 tx=0x7fe770004e80 comp rx=0 tx=0).stop 2023-08-25T02:25:56.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 shutdown_connections 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a25e10 0x5617b5a248d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21880 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe756ffd700 0 client.15045 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a10980 0x5617b5a1a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1545789016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a112e0 0x5617b5a1a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 >> 172.21.15.89:0/1545789016 conn(0x5617b5985090 msgr2=0x5617b5a05af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 shutdown_connections 2023-08-25T02:25:56.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 1 -- 172.21.15.89:0/1545789016 wait complete. 2023-08-25T02:25:56.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.437+0000 7fe789f87b80 20 client.15045 trim_cache size 0 max 16384 2023-08-25T02:25:56.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.454+0000 7fe789f87b80 1 Processor -- start 2023-08-25T02:25:56.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.454+0000 7fe789f87b80 1 -- start start 2023-08-25T02:25:56.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a04ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 0x5617b5a14b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b57e48d0 con 0x5617b5a112e0 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5a0c700 con 0x5617b5a10980 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe789f87b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5922df0 con 0x5617b5a11d00 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47510/0 (socket says 172.21.15.89:47510) 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 -- 172.21.15.89:0/410139572 learned_addr learned my addr 172.21.15.89:0/410139572 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe7777fe700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a04ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe777fff700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 0x5617b5a14b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 -- 172.21.15.89:0/410139572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 msgr2=0x5617b5a14b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 0x5617b5a14b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 -- 172.21.15.89:0/410139572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 msgr2=0x5617b5a04ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a04ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 -- 172.21.15.89:0/410139572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617b5777c10 con 0x5617b5a112e0 2023-08-25T02:25:56.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe777fff700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 0x5617b5a14b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe776ffd700 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe76800edd0 tx=0x7fe768009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83a76430b6e38b6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe75e7fc700 1 -- 172.21.15.89:0/410139572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768011c90 con 0x5617b5a112e0 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe75e7fc700 1 -- 172.21.15.89:0/410139572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe768011df0 con 0x5617b5a112e0 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.455+0000 7fe75e7fc700 1 -- 172.21.15.89:0/410139572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768010780 con 0x5617b5a112e0 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- 172.21.15.89:0/410139572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 msgr2=0x5617b5a217a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe76800edd0 tx=0x7fe768009f90 comp rx=0 tx=0).stop 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- 172.21.15.89:0/410139572 shutdown_connections 2023-08-25T02:25:56.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a04ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a11d00 0x5617b5a14b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 --2- 172.21.15.89:0/410139572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a112e0 0x5617b5a217a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- 172.21.15.89:0/410139572 >> 172.21.15.89:0/410139572 conn(0x5617b5985090 msgr2=0x5617b5a19090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- 172.21.15.89:0/410139572 shutdown_connections 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- 172.21.15.89:0/410139572 wait complete. 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 -- start start 2023-08-25T02:25:56.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.456+0000 7fe789f87b80 1 Processor -- start 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a1e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a20d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5a218b0 con 0x5617b5a11d00 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5a21a20 con 0x5617b5a10980 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b5a79800 con 0x5617b5a112e0 2023-08-25T02:25:56.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 10 client.0 _ll_register_callbacks cb 0x7fe76c095e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47518/0 (socket says 172.21.15.89:47518) 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 -- 172.21.15.89:0/1386297620 learned_addr learned my addr 172.21.15.89:0/1386297620 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a1e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe776ffd700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a20d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 -- 172.21.15.89:0/1386297620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 msgr2=0x5617b5a20d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a20d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 -- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 msgr2=0x5617b5a1e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:25:56.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a1e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 -- 172.21.15.89:0/1386297620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76800ea00 con 0x5617b5a11d00 2023-08-25T02:25:56.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe777fff700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe770002230 tx=0x7fe7700049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87adf9c93f472bf server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:56.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe756ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:25:56.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770018030 con 0x5617b5a11d00 2023-08-25T02:25:56.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe770004000 con 0x5617b5a11d00 2023-08-25T02:25:56.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.457+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617b5a03860 con 0x5617b5a11d00 2023-08-25T02:25:56.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe770014c60 con 0x5617b5a11d00 2023-08-25T02:25:56.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 10 client.15051 fetch_fsmap learned FSMap version 22 2023-08-25T02:25:56.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5617b57b5200 con 0x5617b5a11d00 2023-08-25T02:25:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5617b5a03860 con 0x5617b5a11d00 2023-08-25T02:25:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe770007b80 con 0x5617b5a11d00 2023-08-25T02:25:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 10 client.15051 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:25:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 10 client.15051 resolve_mds: resolved name 'b' to daemon mds.b 2023-08-25T02:25:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 20 client.15051 populate_metadata read hostname 'smithi089' 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a7e680 0x5617b5a7ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 4 client.15051 mds_command: new command op to 24509 tid=0[{"prefix": "status"}] 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5617b595b920 con 0x5617b5a7e680 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.458+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a7e680 0x5617b5a7ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe756ffd700 10 client.15051 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe7777fe700 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a7e680 0x5617b5a7ab00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fe9bd579590f8b server_cookie=f5c555a685e2c3dd in_seq=0 out_seq=0 2023-08-25T02:25:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe756ffd700 1 -- 172.21.15.89:0/1386297620 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5617b595b920 con 0x5617b5a7e680 2023-08-25T02:25:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe756ffd700 10 client.15051 handle_command_reply: tid=0 2023-08-25T02:25:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 client.15051 shutdown 2023-08-25T02:25:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 msgr2=0x5617b5a21260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe770002230 tx=0x7fe7700049a0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 shutdown_connections 2023-08-25T02:25:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5617b5a7e680 0x5617b5a7ab00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5617b5a10980 0x5617b5a1e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe756ffd700 0 client.15051 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:25:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5617b5a11d00 0x5617b5a21260 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 --2- 172.21.15.89:0/1386297620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617b5a112e0 0x5617b5a20d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 >> 172.21.15.89:0/1386297620 conn(0x5617b5985090 msgr2=0x5617b5a19090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 shutdown_connections 2023-08-25T02:25:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 1 -- 172.21.15.89:0/1386297620 wait complete. 2023-08-25T02:25:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.459+0000 7fe789f87b80 20 client.15051 trim_cache size 0 max 16384 2023-08-25T02:25:56.476 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24509, 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:25:56.477 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:25:56.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 51.858804141999997, 2023-08-25T02:25:56.478 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:25:56.479 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 80, 2023-08-25T02:25:56.479 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:25:56.479 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 57.078296631999997 2023-08-25T02:25:56.480 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:25:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 msgr2=0x7fe7600a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 0x7fe7600a2410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe768005fd0 tx=0x7fe768005ee0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 msgr2=0x7fe778196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009fd0 tx=0x7fe76c00f740 comp rx=0 tx=0).stop 2023-08-25T02:25:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 shutdown_connections 2023-08-25T02:25:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe778100b10 0x7fe77819dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7780fef80 0x7fe778196660 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fe76009ff50 0x7fe7600a2410 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 --2- 172.21.15.89:0/1387004752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe778100170 0x7fe778196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 >> 172.21.15.89:0/1387004752 conn(0x7fe7780fa4f0 msgr2=0x7fe778107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 shutdown_connections 2023-08-25T02:25:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:56.460+0000 7fe77dde3700 1 -- 172.21.15.89:0/1387004752 wait complete. 2023-08-25T02:25:56.484 INFO:tasks.cephadm:Setup complete, yielding 2023-08-25T02:25:56.485 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-08-25T02:25:56.501 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:25:56.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph orch status' 2023-08-25T02:25:56.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:56 smithi089 ceph-mon[98732]: pgmap v276: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:56.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/171433743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:56.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:56 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1834914130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:56.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:56 smithi121 ceph-mon[100099]: pgmap v276: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:56.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:56 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/171433743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:56.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:56 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1834914130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:56.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:56 smithi122 ceph-mon[100457]: pgmap v276: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:25:56.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:56 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/171433743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:25:56.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:56 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1834914130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:25:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:58.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 -- 172.21.15.89:0/584906608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc101be0 msgr2=0x7fbefc102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 --2- 172.21.15.89:0/584906608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc101be0 0x7fbefc102040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbef80099c0 tx=0x7fbef8009cd0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 -- 172.21.15.89:0/584906608 shutdown_connections 2023-08-25T02:25:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 --2- 172.21.15.89:0/584906608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc1009e0 0x7fbefc100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 --2- 172.21.15.89:0/584906608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 --2- 172.21.15.89:0/584906608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc101be0 0x7fbefc102040 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 -- 172.21.15.89:0/584906608 >> 172.21.15.89:0/584906608 conn(0x7fbefc0fbf40 msgr2=0x7fbefc0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 -- 172.21.15.89:0/584906608 shutdown_connections 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.244+0000 7fbf03615700 1 -- 172.21.15.89:0/584906608 wait complete. 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 Processor -- start 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 -- start start 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 0x7fbefc072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc0739d0 con 0x7fbefc1009e0 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc073b40 con 0x7fbefc101be0 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf03615700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc1a0790 con 0x7fbefc102580 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf013b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf013b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47528/0 (socket says 172.21.15.89:47528) 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf013b1700 1 -- 172.21.15.89:0/2654802345 learned_addr learned my addr 172.21.15.89:0/2654802345 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.245+0000 7fbf00bb0700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 0x7fbefc072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 -- 172.21.15.89:0/2654802345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 msgr2=0x7fbefc073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf01bb2700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 -- 172.21.15.89:0/2654802345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 msgr2=0x7fbefc072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:58.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 0x7fbefc072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 -- 172.21.15.89:0/2654802345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80095f0 con 0x7fbefc1009e0 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf00bb0700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 0x7fbefc072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf01bb2700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf013b1700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c970 tx=0x7fbeec00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004220 con 0x7fbefc1009e0 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeec007d60 con 0x7fbefc1009e0 2023-08-25T02:25:58.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc1a0a70 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec003d00 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.246+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc1a0f70 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.247+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbefc10c2b0 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.248+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fbeec004380 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.248+0000 7fbef27fc700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 0x7fbee80a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.249+0000 7fbf00bb0700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 0x7fbee80a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.249+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbeec0cb0e0 con 0x7fbefc1009e0 2023-08-25T02:25:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.249+0000 7fbf00bb0700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 0x7fbee80a2490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8004520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:25:58.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.253+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fbeec094f20 con 0x7fbefc1009e0 2023-08-25T02:25:58.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.458+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbefc060d60 con 0x7fbee809ffd0 2023-08-25T02:25:58.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.459+0000 7fbef27fc700 1 -- 172.21.15.89:0/2654802345 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbefc060d60 con 0x7fbee809ffd0 2023-08-25T02:25:58.461 INFO:teuthology.orchestra.run.smithi089.stdout:Backend: cephadm 2023-08-25T02:25:58.461 INFO:teuthology.orchestra.run.smithi089.stdout:Available: Yes 2023-08-25T02:25:58.461 INFO:teuthology.orchestra.run.smithi089.stdout:Paused: No 2023-08-25T02:25:58.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 msgr2=0x7fbee80a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 0x7fbee80a2490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8004520 comp rx=0 tx=0).stop 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 msgr2=0x7fbefc0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c970 tx=0x7fbeec00cd30 comp rx=0 tx=0).stop 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 shutdown_connections 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbefc102580 0x7fbefc073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbefc1009e0 0x7fbefc0746c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbee809ffd0 0x7fbee80a2490 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 --2- 172.21.15.89:0/2654802345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbefc101be0 0x7fbefc072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:25:58.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 >> 172.21.15.89:0/2654802345 conn(0x7fbefc0fbf40 msgr2=0x7fbefc0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:25:58.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 shutdown_connections 2023-08-25T02:25:58.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:25:58.462+0000 7fbf03615700 1 -- 172.21.15.89:0/2654802345 wait complete. 2023-08-25T02:25:58.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:58 smithi121 ceph-mon[100099]: pgmap v277: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:58.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:58 smithi122 ceph-mon[100457]: pgmap v277: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:58.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:58 smithi089 ceph-mon[98732]: pgmap v277: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:25:58.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph orch ps' 2023-08-25T02:25:59.461 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:25:59.648 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:25:59 smithi089 ceph-mon[98732]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:25:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:25:59 smithi121 ceph-mon[100099]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:25:59.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:25:59 smithi122 ceph-mon[100457]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:00.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:00 smithi089 ceph-mon[98732]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 -- 172.21.15.89:0/4207588961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2041009b0 msgr2=0x7fa204100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 --2- 172.21.15.89:0/4207588961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2041009b0 0x7fa204100dd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40099c0 tx=0x7fa1f4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 -- 172.21.15.89:0/4207588961 shutdown_connections 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 --2- 172.21.15.89:0/4207588961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204102550 0x7fa204108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 --2- 172.21.15.89:0/4207588961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2041009b0 0x7fa204100dd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 --2- 172.21.15.89:0/4207588961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa204101bb0 0x7fa204102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 -- 172.21.15.89:0/4207588961 >> 172.21.15.89:0/4207588961 conn(0x7fa2040fbf10 msgr2=0x7fa2040fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 -- 172.21.15.89:0/4207588961 shutdown_connections 2023-08-25T02:26:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.656+0000 7fa20a60a700 1 -- 172.21.15.89:0/4207588961 wait complete. 2023-08-25T02:26:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 Processor -- start 2023-08-25T02:26:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 -- start start 2023-08-25T02:26:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2041009b0 0x7fa2040746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204102550 0x7fa204073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040739d0 con 0x7fa204101bb0 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204073b40 con 0x7fa204102550 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa20a60a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2041a0780 con 0x7fa2041009b0 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47544/0 (socket says 172.21.15.89:47544) 2023-08-25T02:26:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 -- 172.21.15.89:0/3260414500 learned_addr learned my addr 172.21.15.89:0/3260414500 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa203fff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2041009b0 0x7fa2040746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 -- 172.21.15.89:0/3260414500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2041009b0 msgr2=0x7fa2040746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2041009b0 0x7fa2040746c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 -- 172.21.15.89:0/3260414500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204102550 msgr2=0x7fa204073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204102550 0x7fa204073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.657+0000 7fa1fbfff700 1 -- 172.21.15.89:0/3260414500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40095f0 con 0x7fa204101bb0 2023-08-25T02:26:00.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa1fbfff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00b530 tx=0x7fa1ec00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:00.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec011830 con 0x7fa204101bb0 2023-08-25T02:26:00.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1ec011e50 con 0x7fa204101bb0 2023-08-25T02:26:00.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2041a0a60 con 0x7fa204101bb0 2023-08-25T02:26:00.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec00f2e0 con 0x7fa204101bb0 2023-08-25T02:26:00.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.658+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2041a0fb0 con 0x7fa204101bb0 2023-08-25T02:26:00.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.659+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa204107940 con 0x7fa204101bb0 2023-08-25T02:26:00.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.660+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fa1ec010580 con 0x7fa204101bb0 2023-08-25T02:26:00.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.661+0000 7fa201ffb700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 0x7fa1f00a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:00.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.661+0000 7fa203fff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 0x7fa1f00a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:00.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.661+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1ec0cb160 con 0x7fa204101bb0 2023-08-25T02:26:00.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.661+0000 7fa203fff700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 0x7fa1f00a2490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40098e0 tx=0x7fa1f400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:00.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.664+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa1ec095040 con 0x7fa204101bb0 2023-08-25T02:26:00.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:00 smithi121 ceph-mon[100099]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:00 smithi122 ceph-mon[100457]: pgmap v278: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:00.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.863+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa20402cdf0 con 0x7fa1f009ffd0 2023-08-25T02:26:00.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.875+0000 7fa201ffb700 1 -- 172.21.15.89:0/3260414500 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fa20402cdf0 con 0x7fa1f009ffd0 2023-08-25T02:26:00.877 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-08-25T02:26:00.878 INFO:teuthology.orchestra.run.smithi089.stdout:mds.a smithi089 running (64s) - 63s 15.7M - 18.0.0-5703-gba6b39e4 95c9c69e1344 2a26d801e41e 2023-08-25T02:26:00.878 INFO:teuthology.orchestra.run.smithi089.stdout:mds.b smithi121 running (61s) - 61s 16.8M - 18.0.0-5703-gba6b39e4 95c9c69e1344 8e673ec31478 2023-08-25T02:26:00.878 INFO:teuthology.orchestra.run.smithi089.stdout:mds.c smithi122 running (72s) - 72s 14.2M - 18.0.0-5703-gba6b39e4 95c9c69e1344 85c3f14932db 2023-08-25T02:26:00.879 INFO:teuthology.orchestra.run.smithi089.stdout:mds.d smithi089 running (79s) - 79s 16.6M - 18.0.0-5703-gba6b39e4 95c9c69e1344 3fe4c700567c 2023-08-25T02:26:00.879 INFO:teuthology.orchestra.run.smithi089.stdout:mds.e smithi121 running (77s) - 77s 14.8M - 18.0.0-5703-gba6b39e4 95c9c69e1344 c8768201ed8b 2023-08-25T02:26:00.879 INFO:teuthology.orchestra.run.smithi089.stdout:mds.f smithi122 running (75s) - 74s 16.9M - 18.0.0-5703-gba6b39e4 95c9c69e1344 a289a00d90b3 2023-08-25T02:26:00.879 INFO:teuthology.orchestra.run.smithi089.stdout:mds.g smithi089 running (66s) - 65s 16.7M - 18.0.0-5703-gba6b39e4 95c9c69e1344 eb15087f785e 2023-08-25T02:26:00.880 INFO:teuthology.orchestra.run.smithi089.stdout:mds.h smithi121 running (70s) - 70s 14.4M - 18.0.0-5703-gba6b39e4 95c9c69e1344 cb8e5d064163 2023-08-25T02:26:00.880 INFO:teuthology.orchestra.run.smithi089.stdout:mds.i smithi122 running (68s) - 68s 16.8M - 18.0.0-5703-gba6b39e4 95c9c69e1344 3ed57c59e61e 2023-08-25T02:26:00.880 INFO:teuthology.orchestra.run.smithi089.stdout:mds.j smithi089 running (82s) - 81s 14.4M - 18.0.0-5703-gba6b39e4 95c9c69e1344 09d18e7bf1d3 2023-08-25T02:26:00.881 INFO:teuthology.orchestra.run.smithi089.stdout:mds.k smithi121 running (84s) - 83s 16.9M - 18.0.0-5703-gba6b39e4 95c9c69e1344 d22661d9f62e 2023-08-25T02:26:00.881 INFO:teuthology.orchestra.run.smithi089.stdout:mds.l smithi122 running (86s) - 85s 16.5M - 18.0.0-5703-gba6b39e4 95c9c69e1344 6295cb31265c 2023-08-25T02:26:00.881 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.x smithi089 *:9283,8765 running (7m) - 7m 515M - 18.0.0-5703-gba6b39e4 95c9c69e1344 90e8ce7beaf3 2023-08-25T02:26:00.881 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.y smithi121 *:8443,8765 running (5m) - 5m 451M - 18.0.0-5703-gba6b39e4 95c9c69e1344 e66daba24690 2023-08-25T02:26:00.882 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.z smithi122 *:8443,8765 running (6m) - 5m 451M - 18.0.0-5703-gba6b39e4 95c9c69e1344 1dd3fb09857e 2023-08-25T02:26:00.882 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a smithi089 running (7m) - 7m 83.7M 2048M 18.0.0-5703-gba6b39e4 95c9c69e1344 4e5b0a3428c1 2023-08-25T02:26:00.882 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b smithi121 running (6m) - 6m 57.5M 2048M 18.0.0-5703-gba6b39e4 95c9c69e1344 da94d094ec01 2023-08-25T02:26:00.883 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c smithi122 running (6m) - 6m 59.8M 2048M 18.0.0-5703-gba6b39e4 95c9c69e1344 ab4abb606932 2023-08-25T02:26:00.883 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (5m) - 5m 57.4M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 3b1ca3c261bd 2023-08-25T02:26:00.884 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi121 running (5m) - 5m 63.4M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 6d361bc38e97 2023-08-25T02:26:00.884 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi122 running (4m) - 4m 58.4M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 c42ea4369036 2023-08-25T02:26:00.884 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi089 running (4m) - 4m 53.7M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 621c00672833 2023-08-25T02:26:00.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi121 running (4m) - 4m 51.9M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 b0ecbeaa3407 2023-08-25T02:26:00.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi122 running (3m) - 3m 53.9M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 8fd7b7d99151 2023-08-25T02:26:00.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 smithi089 running (3m) - 3m 50.0M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 435c2cc2f890 2023-08-25T02:26:00.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 smithi121 running (3m) - 3m 54.7M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 ac92374ba107 2023-08-25T02:26:00.886 INFO:teuthology.orchestra.run.smithi089.stdout:osd.8 smithi122 running (2m) - 2m 52.6M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 bfbddedf7032 2023-08-25T02:26:00.887 INFO:teuthology.orchestra.run.smithi089.stdout:osd.9 smithi089 running (2m) - 2m 49.0M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 db9ed0a9563a 2023-08-25T02:26:00.887 INFO:teuthology.orchestra.run.smithi089.stdout:osd.10 smithi121 running (2m) - 2m 52.6M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 07ffcb33577f 2023-08-25T02:26:00.887 INFO:teuthology.orchestra.run.smithi089.stdout:osd.11 smithi122 running (104s) - 104s 45.9M 4292M 18.0.0-5703-gba6b39e4 95c9c69e1344 4291bdcaa8bf 2023-08-25T02:26:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 msgr2=0x7fa1f00a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:00.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 0x7fa1f00a2490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40098e0 tx=0x7fa1f400b540 comp rx=0 tx=0).stop 2023-08-25T02:26:00.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 msgr2=0x7fa204072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:00.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00b530 tx=0x7fa1ec00b8f0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 shutdown_connections 2023-08-25T02:26:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fa1f009ffd0 0x7fa1f00a2490 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204102550 0x7fa204073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2041009b0 0x7fa2040746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 --2- 172.21.15.89:0/3260414500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa204101bb0 0x7fa204072d10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:00.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 >> 172.21.15.89:0/3260414500 conn(0x7fa2040fbf10 msgr2=0x7fa2040fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:00.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 shutdown_connections 2023-08-25T02:26:00.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:00.878+0000 7fa20a60a700 1 -- 172.21.15.89:0/3260414500 wait complete. 2023-08-25T02:26:01.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph orch ls' 2023-08-25T02:26:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:02.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:02 smithi121 ceph-mon[100099]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:02.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:02 smithi121 ceph-mon[100099]: pgmap v279: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:26:02.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:02 smithi122 ceph-mon[100457]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:02.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:02 smithi122 ceph-mon[100457]: pgmap v279: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:26:02.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:02 smithi089 ceph-mon[98732]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:02 smithi089 ceph-mon[98732]: pgmap v279: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-08-25T02:26:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 -- 172.21.15.89:0/1559643510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc106700 msgr2=0x7f6dfc108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 --2- 172.21.15.89:0/1559643510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc106700 0x7f6dfc108b10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009990 tx=0x7f6dec009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 -- 172.21.15.89:0/1559643510 shutdown_connections 2023-08-25T02:26:03.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 --2- 172.21.15.89:0/1559643510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 --2- 172.21.15.89:0/1559643510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc106700 0x7f6dfc108b10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 --2- 172.21.15.89:0/1559643510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc1061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 -- 172.21.15.89:0/1559643510 >> 172.21.15.89:0/1559643510 conn(0x7f6dfc0fae30 msgr2=0x7f6dfc0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 -- 172.21.15.89:0/1559643510 shutdown_connections 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 -- 172.21.15.89:0/1559643510 wait complete. 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.060+0000 7f6e01392700 1 Processor -- start 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 -- start start 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 0x7f6dfc19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc19e140 con 0x7f6dfc1014a0 2023-08-25T02:26:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc19e280 con 0x7f6dfc106700 2023-08-25T02:26:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6e01392700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc19e410 con 0x7f6dfc103dd0 2023-08-25T02:26:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47556/0 (socket says 172.21.15.89:47556) 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 -- 172.21.15.89:0/2954108120 learned_addr learned my addr 172.21.15.89:0/2954108120 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfb7fe700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 0x7f6dfc19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 -- 172.21.15.89:0/2954108120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 msgr2=0x7f6dfc196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfa7fc700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 -- 172.21.15.89:0/2954108120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 msgr2=0x7f6dfc19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 0x7f6dfc19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfaffd700 1 -- 172.21.15.89:0/2954108120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0095f0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfb7fe700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 0x7f6dfc19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.061+0000 7f6dfa7fc700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6dfaffd700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6df4007e50 tx=0x7f6df400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df400adc0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc19e6f0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6df40128e0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40117a0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.062+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc19ec40 con 0x7f6dfc1014a0 2023-08-25T02:26:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.063+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc0fc9b0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.064+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6df4012cc0 con 0x7f6dfc1014a0 2023-08-25T02:26:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.064+0000 7f6de3fff700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 0x7f6de40a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.064+0000 7f6dfa7fc700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 0x7f6de40a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.064+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6df40cb230 con 0x7f6dfc1014a0 2023-08-25T02:26:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.065+0000 7f6dfa7fc700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 0x7f6de40a2410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6df0007900 tx=0x7f6df0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.069+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6df4095110 con 0x7f6dfc1014a0 2023-08-25T02:26:03.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.266+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dfc02cce0 con 0x7f6de409ff50 2023-08-25T02:26:03.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.271+0000 7f6de3fff700 1 -- 172.21.15.89:0/2954108120 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f6dfc02cce0 con 0x7f6de409ff50 2023-08-25T02:26:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-08-25T02:26:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:mds.all 12/12 - 90s smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2023-08-25T02:26:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:mgr 3/3 - 6m smithi089=x;smithi121=y;smithi122=z;count:3 2023-08-25T02:26:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:mon 3/3 - 6m smithi089:172.21.15.89=a;smithi121:172.21.15.121=b;smithi122:172.21.15.122=c;count:3 2023-08-25T02:26:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:osd 12 - - 2023-08-25T02:26:03.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 msgr2=0x7f6de40a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:03.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 0x7f6de40a2410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6df0007900 tx=0x7f6df0008040 comp rx=0 tx=0).stop 2023-08-25T02:26:03.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 msgr2=0x7f6dfc1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:03.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6df4007e50 tx=0x7f6df400a340 comp rx=0 tx=0).stop 2023-08-25T02:26:03.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 shutdown_connections 2023-08-25T02:26:03.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6de409ff50 0x7f6de40a2410 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dfc106700 0x7f6dfc19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6dfc1014a0 0x7f6dfc1965b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 --2- 172.21.15.89:0/2954108120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dfc103dd0 0x7f6dfc196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:03.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 >> 172.21.15.89:0/2954108120 conn(0x7f6dfc0fae30 msgr2=0x7f6dfc102110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:03.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 shutdown_connections 2023-08-25T02:26:03.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:03.274+0000 7f6e01392700 1 -- 172.21.15.89:0/2954108120 wait complete. 2023-08-25T02:26:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-08-25T02:26:03.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-08-25T02:26:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2023-08-25T02:26:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-08-25T02:26:03.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:03 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2023-08-25T02:26:03.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-08-25T02:26:03.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2023-08-25T02:26:03.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-08-25T02:26:03.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:03 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2023-08-25T02:26:03.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph orch host ls' 2023-08-25T02:26:03.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-08-25T02:26:03.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2023-08-25T02:26:03.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2023-08-25T02:26:03.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:03 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2023-08-25T02:26:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:04.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:04.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: pgmap v280: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:04.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-08-25T02:26:04.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-08-25T02:26:04.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2023-08-25T02:26:04.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2023-08-25T02:26:04.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:04 smithi089 ceph-mon[98732]: osdmap e81: 12 total, 12 up, 12 in 2023-08-25T02:26:04.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:04.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: pgmap v280: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:04.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-08-25T02:26:04.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-08-25T02:26:04.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2023-08-25T02:26:04.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-08-25T02:26:04.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-08-25T02:26:04.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-08-25T02:26:04.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-08-25T02:26:04.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2023-08-25T02:26:04.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-08-25T02:26:04.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2023-08-25T02:26:04.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:04 smithi121 ceph-mon[100099]: osdmap e81: 12 total, 12 up, 12 in 2023-08-25T02:26:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: pgmap v280: 97 pgs: 97 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-08-25T02:26:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-08-25T02:26:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-08-25T02:26:04.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2023-08-25T02:26:04.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2023-08-25T02:26:04.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:04 smithi122 ceph-mon[100457]: osdmap e81: 12 total, 12 up, 12 in 2023-08-25T02:26:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 -- 172.21.15.89:0/311891726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078101bd0 msgr2=0x7ff078102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 --2- 172.21.15.89:0/311891726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078101bd0 0x7ff078102030 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff0640099c0 tx=0x7ff064009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 -- 172.21.15.89:0/311891726 shutdown_connections 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 --2- 172.21.15.89:0/311891726 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0781009d0 0x7ff078100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 --2- 172.21.15.89:0/311891726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff078102570 0x7ff078108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 --2- 172.21.15.89:0/311891726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078101bd0 0x7ff078102030 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 -- 172.21.15.89:0/311891726 >> 172.21.15.89:0/311891726 conn(0x7ff0780fbf30 msgr2=0x7ff0780fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 -- 172.21.15.89:0/311891726 shutdown_connections 2023-08-25T02:26:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.452+0000 7ff07ed53700 1 -- 172.21.15.89:0/311891726 wait complete. 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 Processor -- start 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 -- start start 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0781009d0 0x7ff0781966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 0x7ff078196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819e270 con 0x7ff078102570 2023-08-25T02:26:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819e3b0 con 0x7ff078101bd0 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07ed53700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819e540 con 0x7ff0781009d0 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47574/0 (socket says 172.21.15.89:47574) 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 -- 172.21.15.89:0/770065521 learned_addr learned my addr 172.21.15.89:0/770065521 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff077fff700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 0x7ff078196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07caef700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0781009d0 0x7ff0781966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 -- 172.21.15.89:0/770065521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0781009d0 msgr2=0x7ff0781966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0781009d0 0x7ff0781966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 -- 172.21.15.89:0/770065521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 msgr2=0x7ff078196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 0x7ff078196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff07d2f0700 1 -- 172.21.15.89:0/770065521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640095f0 con 0x7ff078102570 2023-08-25T02:26:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.453+0000 7ff077fff700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 0x7ff078196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff07d2f0700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff07000b9a0 tx=0x7ff07000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000c9b0 con 0x7ff078102570 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07819e820 con 0x7ff078102570 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070014440 con 0x7ff078102570 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700075d0 con 0x7ff078102570 2023-08-25T02:26:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.454+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07819ed70 con 0x7ff078102570 2023-08-25T02:26:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.455+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff078107fd0 con 0x7ff078102570 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.456+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7ff0700145b0 con 0x7ff078102570 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.456+0000 7ff075ffb700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 0x7ff0680a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.456+0000 7ff07caef700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 0x7ff0680a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.456+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff070006e40 con 0x7ff078102570 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.457+0000 7ff07caef700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 0x7ff0680a2410 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff078101a30 tx=0x7ff06c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.461+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff070094770 con 0x7ff078102570 2023-08-25T02:26:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:05 smithi089 ceph-mon[98732]: osdmap e82: 12 total, 12 up, 12 in 2023-08-25T02:26:05.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.653+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07802cd60 con 0x7ff06809ff50 2023-08-25T02:26:05.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.655+0000 7ff075ffb700 1 -- 172.21.15.89:0/770065521 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7ff07802cd60 con 0x7ff06809ff50 2023-08-25T02:26:05.656 INFO:teuthology.orchestra.run.smithi089.stdout:HOST ADDR LABELS STATUS 2023-08-25T02:26:05.656 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 172.21.15.89 2023-08-25T02:26:05.656 INFO:teuthology.orchestra.run.smithi089.stdout:smithi121 172.21.15.121 2023-08-25T02:26:05.657 INFO:teuthology.orchestra.run.smithi089.stdout:smithi122 172.21.15.122 2023-08-25T02:26:05.657 INFO:teuthology.orchestra.run.smithi089.stdout:3 hosts in cluster 2023-08-25T02:26:05.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 msgr2=0x7ff0680a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:05.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 0x7ff0680a2410 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff078101a30 tx=0x7ff06c005cf0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 msgr2=0x7ff07819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff07000b9a0 tx=0x7ff07000bd60 comp rx=0 tx=0).stop 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 shutdown_connections 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078102570 0x7ff07819dca0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0781009d0 0x7ff0781966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06809ff50 0x7ff0680a2410 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 --2- 172.21.15.89:0/770065521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff078101bd0 0x7ff078196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 >> 172.21.15.89:0/770065521 conn(0x7ff0780fbf30 msgr2=0x7ff078105110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 shutdown_connections 2023-08-25T02:26:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:05.657+0000 7ff07ed53700 1 -- 172.21.15.89:0/770065521 wait complete. 2023-08-25T02:26:05.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:05 smithi121 ceph-mon[100099]: osdmap e82: 12 total, 12 up, 12 in 2023-08-25T02:26:05.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:05 smithi122 ceph-mon[100457]: osdmap e82: 12 total, 12 up, 12 in 2023-08-25T02:26:06.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph orch device ls' 2023-08-25T02:26:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:06.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:06 smithi121 ceph-mon[100099]: pgmap v283: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-08-25T02:26:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:06 smithi121 ceph-mon[100099]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:06 smithi121 ceph-mon[100099]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-08-25T02:26:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:06 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:06.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:06 smithi122 ceph-mon[100457]: pgmap v283: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-08-25T02:26:06.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:06 smithi122 ceph-mon[100457]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:06.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:06 smithi122 ceph-mon[100457]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-08-25T02:26:06.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:06 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:06.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:06 smithi089 ceph-mon[98732]: pgmap v283: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-08-25T02:26:06.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:06 smithi089 ceph-mon[98732]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:06.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:06 smithi089 ceph-mon[98732]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-08-25T02:26:06.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:06 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 -- 172.21.15.89:0/3955383443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102690 msgr2=0x7fd05c108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 --2- 172.21.15.89:0/3955383443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102690 0x7fd05c108f40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b0a0 tx=0x7fd05400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 -- 172.21.15.89:0/3955383443 shutdown_connections 2023-08-25T02:26:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 --2- 172.21.15.89:0/3955383443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c0fec70 0x7fd05c0ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 --2- 172.21.15.89:0/3955383443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102690 0x7fd05c108f40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 --2- 172.21.15.89:0/3955383443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c101cf0 0x7fd05c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 -- 172.21.15.89:0/3955383443 >> 172.21.15.89:0/3955383443 conn(0x7fd05c0fa5d0 msgr2=0x7fd05c0fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:07.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 -- 172.21.15.89:0/3955383443 shutdown_connections 2023-08-25T02:26:07.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.822+0000 7fd060d64700 1 -- 172.21.15.89:0/3955383443 wait complete. 2023-08-25T02:26:07.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.823+0000 7fd060d64700 1 Processor -- start 2023-08-25T02:26:07.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.823+0000 7fd060d64700 1 -- start start 2023-08-25T02:26:07.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 0x7fd05c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 0x7fd05c196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c102690 0x7fd05c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c19e320 con 0x7fd05c101cf0 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c19e460 con 0x7fd05c102690 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd060d64700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c19e5f0 con 0x7fd05c0fec70 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd059d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 0x7fd05c196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd059d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 0x7fd05c196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47584/0 (socket says 172.21.15.89:47584) 2023-08-25T02:26:07.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd059d9b700 1 -- 172.21.15.89:0/1783471331 learned_addr learned my addr 172.21.15.89:0/1783471331 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:07.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 0x7fd05c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:07.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05ad9d700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c102690 0x7fd05c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:07.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 -- 172.21.15.89:0/1783471331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c102690 msgr2=0x7fd05c19dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:07.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c102690 0x7fd05c19dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 -- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 msgr2=0x7fd05c196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 0x7fd05c196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 -- 172.21.15.89:0/1783471331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054009d00 con 0x7fd05c0fec70 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.824+0000 7fd05a59c700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 0x7fd05c196790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00d750 tx=0x7fd04c00da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.825+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c00f890 con 0x7fd05c0fec70 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.825+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c19e8d0 con 0x7fd05c0fec70 2023-08-25T02:26:07.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.825+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c19ee20 con 0x7fd05c0fec70 2023-08-25T02:26:07.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.825+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd04c010460 con 0x7fd05c0fec70 2023-08-25T02:26:07.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.825+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0195c0 con 0x7fd05c0fec70 2023-08-25T02:26:07.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.826+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c10c1c0 con 0x7fd05c0fec70 2023-08-25T02:26:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.827+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fd04c0105d0 con 0x7fd05c0fec70 2023-08-25T02:26:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.827+0000 7fd04b7fe700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 0x7fd0440a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.827+0000 7fd059d9b700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 0x7fd0440a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.827+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fd04c0cabf0 con 0x7fd05c0fec70 2023-08-25T02:26:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.828+0000 7fd059d9b700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 0x7fd0440a2490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd050009ca0 tx=0x7fd050009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:07.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:07.832+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd04c094960 con 0x7fd05c0fec70 2023-08-25T02:26:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.036+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd05c060d60 con 0x7fd04409ffd0 2023-08-25T02:26:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.038+0000 7fd04b7fe700 1 -- 172.21.15.89:0/1783471331 <== mgr.14152 v2:172.21.15.89:6800/1949193934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7fd05c060d60 con 0x7fd04409ffd0 2023-08-25T02:26:08.040 INFO:teuthology.orchestra.run.smithi089.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-08-25T02:26:08.040 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 2m ago LVM detected, locked 2023-08-25T02:26:08.040 INFO:teuthology.orchestra.run.smithi089.stdout:smithi121 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZF400BGN 372G No 2m ago LVM detected, locked 2023-08-25T02:26:08.040 INFO:teuthology.orchestra.run.smithi089.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 372G No 105s ago LVM detected, locked 2023-08-25T02:26:08.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.040+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 msgr2=0x7fd0440a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.040+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 0x7fd0440a2490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd050009ca0 tx=0x7fd050009260 comp rx=0 tx=0).stop 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.040+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 msgr2=0x7fd05c196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.040+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 0x7fd05c196790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00d750 tx=0x7fd04c00da60 comp rx=0 tx=0).stop 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 shutdown_connections 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fd04409ffd0 0x7fd0440a2490 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd05c102690 0x7fd05c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd05c0fec70 0x7fd05c196790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 --2- 172.21.15.89:0/1783471331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c101cf0 0x7fd05c196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 >> 172.21.15.89:0/1783471331 conn(0x7fd05c0fa5d0 msgr2=0x7fd05c0fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 shutdown_connections 2023-08-25T02:26:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:08.041+0000 7fd060d64700 1 -- 172.21.15.89:0/1783471331 wait complete. 2023-08-25T02:26:08.435 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-08-25T02:26:08.447 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:26:08.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph fs dump' 2023-08-25T02:26:08.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:08 smithi121 ceph-mon[100099]: pgmap v284: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-08-25T02:26:08.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:08 smithi122 ceph-mon[100457]: pgmap v284: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-08-25T02:26:08.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:08 smithi089 ceph-mon[98732]: pgmap v284: 97 pgs: 3 peering, 94 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-08-25T02:26:09.004 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:09.524 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:09 smithi089 ceph-mon[98732]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:09.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:09 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:09 smithi121 ceph-mon[100099]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:09 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:09 smithi122 ceph-mon[100457]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:09 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- 172.21.15.89:0/2322378293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab81066a0 msgr2=0x7fdab8108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- 172.21.15.89:0/2322378293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab81066a0 0x7fdab8108b10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdab000b0a0 tx=0x7fdab000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- 172.21.15.89:0/2322378293 shutdown_connections 2023-08-25T02:26:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- 172.21.15.89:0/2322378293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab81066a0 0x7fdab8108b10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- 172.21.15.89:0/2322378293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab8101440 0x7fdab8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- 172.21.15.89:0/2322378293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab8106160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- 172.21.15.89:0/2322378293 >> 172.21.15.89:0/2322378293 conn(0x7fdab80fae30 msgr2=0x7fdab80fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- 172.21.15.89:0/2322378293 shutdown_connections 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- 172.21.15.89:0/2322378293 wait complete. 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 Processor -- start 2023-08-25T02:26:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 -- start start 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.191+0000 7fdac0010700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab80ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 0x7fdab80ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab81000b0 con 0x7fdab8101440 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab8100220 con 0x7fdab8103d70 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab81a07e0 con 0x7fdab81066a0 2023-08-25T02:26:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47592/0 (socket says 172.21.15.89:47592) 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 -- 172.21.15.89:0/2162379558 learned_addr learned my addr 172.21.15.89:0/2162379558 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabe5ad700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 0x7fdab80ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabd5ab700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab80ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 -- 172.21.15.89:0/2162379558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 msgr2=0x7fdab80ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 0x7fdab80ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 -- 172.21.15.89:0/2162379558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 msgr2=0x7fdab80ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab80ff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 -- 172.21.15.89:0/2162379558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009d00 con 0x7fdab8101440 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabd5ab700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab80ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabe5ad700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 0x7fdab80ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdabddac700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdab400b530 tx=0x7fdab400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:10.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4011830 con 0x7fdab8101440 2023-08-25T02:26:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdab4011e50 con 0x7fdab8101440 2023-08-25T02:26:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab81a0ac0 con 0x7fdab8101440 2023-08-25T02:26:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab400f2e0 con 0x7fdab8101440 2023-08-25T02:26:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.192+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab81a1010 con 0x7fdab8101440 2023-08-25T02:26:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.193+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab80fc9b0 con 0x7fdab8101440 2023-08-25T02:26:10.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.194+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fdab4010580 con 0x7fdab8101440 2023-08-25T02:26:10.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.195+0000 7fdaaeffd700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 0x7fdaa40a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:10.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.195+0000 7fdabd5ab700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 0x7fdaa40a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:10.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.195+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fdab40cb270 con 0x7fdab8101440 2023-08-25T02:26:10.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.195+0000 7fdabd5ab700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 0x7fdaa40a2490 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8005fd0 tx=0x7fdaa8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:10.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.199+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fdab4094f80 con 0x7fdab8101440 2023-08-25T02:26:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.434+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdab810c220 con 0x7fdab8101440 2023-08-25T02:26:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.435+0000 7fdaaeffd700 1 -- 172.21.15.89:0/2162379558 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2945 (secure 0 0 0) 0x7fdab4015ba0 con 0x7fdab8101440 2023-08-25T02:26:10.436 INFO:teuthology.orchestra.run.smithi089.stdout:e22 2023-08-25T02:26:10.436 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-08-25T02:26:10.436 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 22 2023-08-25T02:26:10.437 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:created 2023-08-25T02:25:04.024043+0000 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2023-08-25T02:25:09.635801+0000 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 300 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2023-08-25T02:26:10.438 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:max_xattr_size 65536 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 0 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 3 2023-08-25T02:26:10.439 INFO:teuthology.orchestra.run.smithi089.stdout:in 0,1,2 2023-08-25T02:26:10.440 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24509,1=24475,2=24470} 2023-08-25T02:26:10.440 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2023-08-25T02:26:10.440 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2023-08-25T02:26:10.440 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data disabled 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:bal_rank_mask -1 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 0 2023-08-25T02:26:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b{0:24509} state up:active seq 3 addr [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c{0:24454} state up:standby-replay seq 1 addr [v2:172.21.15.122:6836/2359799361,v1:172.21.15.122:6837/2359799361] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.i{1:24475} state up:active seq 5 addr [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.f{1:24442} state up:standby-replay seq 1 addr [v2:172.21.15.122:6834/3100558972,v1:172.21.15.122:6835/3100558972] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.h{2:24470} state up:active seq 6 addr [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.e{2:24440} state up:standby-replay seq 1 addr [v2:172.21.15.121:6834/217468698,v1:172.21.15.121:6835/217468698] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.442 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:10.443 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:10.443 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2023-08-25T02:26:10.443 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:10.443 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.j{-1:14553} state up:standby seq 1 addr [v2:172.21.15.89:6834/1349178421,v1:172.21.15.89:6835/1349178421] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.444 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d{-1:14559} state up:standby seq 1 addr [v2:172.21.15.89:6836/826072159,v1:172.21.15.89:6837/826072159] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.444 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.g{-1:14619} state up:standby seq 1 addr [v2:172.21.15.89:6838/3717526782,v1:172.21.15.89:6839/3717526782] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.444 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a{-1:14631} state up:standby seq 1 addr [v2:172.21.15.89:6840/3065557297,v1:172.21.15.89:6841/3065557297] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.444 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.122:6832/1630095418,v1:172.21.15.122:6833/1630095418] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.444 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.k{-1:24416} state up:standby seq 1 addr [v2:172.21.15.121:6832/2625413815,v1:172.21.15.121:6833/2625413815] compat {c=[1],r=[1],i=[fff]}] 2023-08-25T02:26:10.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 msgr2=0x7fdaa40a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:10.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 0x7fdaa40a2490 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8005fd0 tx=0x7fdaa8005e30 comp rx=0 tx=0).stop 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 msgr2=0x7fdab8100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdab400b530 tx=0x7fdab400b8f0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 shutdown_connections 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdab81066a0 0x7fdab80ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdab8101440 0x7fdab8100da0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fdaa409ffd0 0x7fdaa40a2490 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 --2- 172.21.15.89:0/2162379558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdab8103d70 0x7fdab80ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:10.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 >> 172.21.15.89:0/2162379558 conn(0x7fdab80fae30 msgr2=0x7fdab81020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:10.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 shutdown_connections 2023-08-25T02:26:10.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:10.437+0000 7fdac0010700 1 -- 172.21.15.89:0/2162379558 wait complete. 2023-08-25T02:26:10.447 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:26:10.761 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: pgmap v285: 97 pgs: 2 peering, 95 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 150 B/s, 0 objects/s recovering 2023-08-25T02:26:10.761 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:10.762 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:10.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2162379558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-08-25T02:26:10.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- bash -c 'ceph osd dump' 2023-08-25T02:26:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: pgmap v285: 97 pgs: 2 peering, 95 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 150 B/s, 0 objects/s recovering 2023-08-25T02:26:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2162379558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: pgmap v285: 97 pgs: 2 peering, 95 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 150 B/s, 0 objects/s recovering 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='mgr.14152 172.21.15.89:0/2119467770' entity='mgr.x' 2023-08-25T02:26:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2162379558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-08-25T02:26:11.313 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:11 smithi089 ceph-mon[98732]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-08-25T02:26:11.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:11 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:26:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:11 smithi121 ceph-mon[100099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-08-25T02:26:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:11 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:26:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:11 smithi122 ceph-mon[100457]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-08-25T02:26:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:11 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:26:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 -- 172.21.15.89:0/3513910207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 msgr2=0x7fbe64100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 --2- 172.21.15.89:0/3513910207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64100f10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbe540099c0 tx=0x7fbe54009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 -- 172.21.15.89:0/3513910207 shutdown_connections 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 --2- 172.21.15.89:0/3513910207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 0x7fbe640ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 --2- 172.21.15.89:0/3513910207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe64107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 --2- 172.21.15.89:0/3513910207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64100f10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 -- 172.21.15.89:0/3513910207 >> 172.21.15.89:0/3513910207 conn(0x7fbe640fae30 msgr2=0x7fbe640fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 -- 172.21.15.89:0/3513910207 shutdown_connections 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.463+0000 7fbe68efa700 1 -- 172.21.15.89:0/3513910207 wait complete. 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 Processor -- start 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 -- start start 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 0x7fbe64198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe6419fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641a0400 con 0x7fbe64100ab0 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641a0540 con 0x7fbe640ff8b0 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe68efa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641a06d0 con 0x7fbe64101450 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe61d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe61d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47608/0 (socket says 172.21.15.89:47608) 2023-08-25T02:26:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe61d9b700 1 -- 172.21.15.89:0/1317352935 learned_addr learned my addr 172.21.15.89:0/1317352935 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.464+0000 7fbe6259c700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 0x7fbe64198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe62d9d700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe6419fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 -- 172.21.15.89:0/1317352935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 msgr2=0x7fbe6419fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe6419fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 -- 172.21.15.89:0/1317352935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 msgr2=0x7fbe64198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 0x7fbe64198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 -- 172.21.15.89:0/1317352935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540095f0 con 0x7fbe64100ab0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe62d9d700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe6419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe61d9b700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004730 tx=0x7fbe54004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5401d070 con 0x7fbe64100ab0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe641a0950 con 0x7fbe64100ab0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe54004110 con 0x7fbe64100ab0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54005290 con 0x7fbe64100ab0 2023-08-25T02:26:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.465+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641a0df0 con 0x7fbe64100ab0 2023-08-25T02:26:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.466+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6410b0b0 con 0x7fbe64100ab0 2023-08-25T02:26:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.467+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fbe54004860 con 0x7fbe64100ab0 2023-08-25T02:26:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.467+0000 7fbe537fe700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 0x7fbe4c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.468+0000 7fbe6259c700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 0x7fbe4c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.468+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbe540cc520 con 0x7fbe64100ab0 2023-08-25T02:26:12.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.468+0000 7fbe6259c700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 0x7fbe4c0a2410 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe64100910 tx=0x7fbe5800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:12.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.471+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fbe54096320 con 0x7fbe64100ab0 2023-08-25T02:26:12.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.660+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fbe6410b2a0 con 0x7fbe64100ab0 2023-08-25T02:26:12.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.661+0000 7fbe537fe700 1 -- 172.21.15.89:0/1317352935 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v82) v1 ==== 56+0+5328 (secure 0 0 0) 0x7fbe54099940 con 0x7fbe64100ab0 2023-08-25T02:26:12.662 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 82 2023-08-25T02:26:12.662 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:26:12.663 INFO:teuthology.orchestra.run.smithi089.stdout:created 2023-08-25T02:18:08.578395+0000 2023-08-25T02:26:12.663 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2023-08-25T02:26:04.395929+0000 2023-08-25T02:26:12.663 INFO:teuthology.orchestra.run.smithi089.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-08-25T02:26:12.663 INFO:teuthology.orchestra.run.smithi089.stdout:crush_version 26 2023-08-25T02:26:12.663 INFO:teuthology.orchestra.run.smithi089.stdout:full_ratio 0.95 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:backfillfull_ratio 0.9 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:nearfull_ratio 0.85 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:require_min_compat_client luminous 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:min_compat_client luminous 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:require_osd_release reef 2023-08-25T02:26:12.664 INFO:teuthology.orchestra.run.smithi089.stdout:stretch_mode_enabled false 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:max_osd 12 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6802/729401604,v1:172.21.15.89:6803/729401604] [v2:172.21.15.89:6804/729401604,v1:172.21.15.89:6805/729401604] exists,up 94583d02-1254-4da8-aae0-95aebc0f9804 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 up in weight 1 up_from 13 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/1465063129,v1:172.21.15.121:6801/1465063129] [v2:172.21.15.121:6802/1465063129,v1:172.21.15.121:6803/1465063129] exists,up 6112d305-65d5-4bdd-b560-8baba3ca0e98 2023-08-25T02:26:12.665 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6800/1354963983,v1:172.21.15.122:6801/1354963983] [v2:172.21.15.122:6802/1354963983,v1:172.21.15.122:6803/1354963983] exists,up e8a1640c-fce8-4ac8-a53f-857213701817 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 up in weight 1 up_from 25 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6810/705728932,v1:172.21.15.89:6811/705728932] [v2:172.21.15.89:6812/705728932,v1:172.21.15.89:6813/705728932] exists,up a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 up in weight 1 up_from 32 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/4228849553,v1:172.21.15.121:6809/4228849553] [v2:172.21.15.121:6810/4228849553,v1:172.21.15.121:6811/4228849553] exists,up f57ae45d-1fb5-41e6-9c04-ba9c235814af 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 up in weight 1 up_from 37 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6808/1949734782,v1:172.21.15.122:6809/1949734782] [v2:172.21.15.122:6810/1949734782,v1:172.21.15.122:6811/1949734782] exists,up a9751b06-a1ca-4fe3-ba70-2dc2733e05b4 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 up in weight 1 up_from 44 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6818/1372241573,v1:172.21.15.89:6819/1372241573] [v2:172.21.15.89:6820/1372241573,v1:172.21.15.89:6821/1372241573] exists,up a77199fb-6a0a-4a5f-83a1-690b403a5791 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 up in weight 1 up_from 49 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6816/62965695,v1:172.21.15.121:6817/62965695] [v2:172.21.15.121:6818/62965695,v1:172.21.15.121:6819/62965695] exists,up f5f512d9-eebe-40fa-beb8-f1e78c6d0af6 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.8 up in weight 1 up_from 56 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6816/1782298471,v1:172.21.15.122:6817/1782298471] [v2:172.21.15.122:6818/1782298471,v1:172.21.15.122:6819/1782298471] exists,up 3417d93a-10fe-4919-9d05-41ef7869e207 2023-08-25T02:26:12.666 INFO:teuthology.orchestra.run.smithi089.stdout:osd.9 up in weight 1 up_from 63 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6826/3342522203,v1:172.21.15.89:6827/3342522203] [v2:172.21.15.89:6828/3342522203,v1:172.21.15.89:6829/3342522203] exists,up 0c4d98e8-6bcc-4ca6-9fec-495733ba95d9 2023-08-25T02:26:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:osd.10 up in weight 1 up_from 68 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/3780412782,v1:172.21.15.121:6825/3780412782] [v2:172.21.15.121:6826/3780412782,v1:172.21.15.121:6827/3780412782] exists,up 8b4dd502-8b3b-4c22-b3c3-79d177e32289 2023-08-25T02:26:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:osd.11 up in weight 1 up_from 75 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6824/2395101680,v1:172.21.15.122:6825/2395101680] [v2:172.21.15.122:6826/2395101680,v1:172.21.15.122:6827/2395101680] exists,up 7e7ae39d-a06b-4a45-9bfd-b969620b53e8 2023-08-25T02:26:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.0 [8,4] 2023-08-25T02:26:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.a [2,11] 2023-08-25T02:26:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.e [2,1] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.1b [2,4] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.1f [7,11] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.1 [4,9] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.e [5,9] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.21 [4,6] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.23 [4,8] 2023-08-25T02:26:12.668 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.29 [3,9] 2023-08-25T02:26:12.669 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6801/3825317465 expires 2023-08-26T02:19:03.221042+0000 2023-08-25T02:26:12.669 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/3529687955 expires 2023-08-26T02:19:03.221042+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6800/3825317465 expires 2023-08-26T02:19:03.221042+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/2328307501 expires 2023-08-26T02:18:40.017976+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6801/730683542 expires 2023-08-26T02:18:40.017976+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/3827846667 expires 2023-08-26T02:18:40.017976+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/4176087350 expires 2023-08-26T02:18:40.017976+0000 2023-08-25T02:26:12.670 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6800/730683542 expires 2023-08-26T02:18:40.017976+0000 2023-08-25T02:26:12.671 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/3259031121 expires 2023-08-26T02:19:03.221042+0000 2023-08-25T02:26:12.671 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/1543179786 expires 2023-08-26T02:19:03.221042+0000 2023-08-25T02:26:12.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 msgr2=0x7fbe4c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 0x7fbe4c0a2410 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe64100910 tx=0x7fbe5800b410 comp rx=0 tx=0).stop 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 msgr2=0x7fbe64198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004730 tx=0x7fbe54004760 comp rx=0 tx=0).stop 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 shutdown_connections 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe4c09ff50 0x7fbe4c0a2410 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe640ff8b0 0x7fbe64198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64101450 0x7fbe6419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 --2- 172.21.15.89:0/1317352935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64100ab0 0x7fbe64198db0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 >> 172.21.15.89:0/1317352935 conn(0x7fbe640fae30 msgr2=0x7fbe640684a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 shutdown_connections 2023-08-25T02:26:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:12.663+0000 7fbe68efa700 1 -- 172.21.15.89:0/1317352935 wait complete. 2023-08-25T02:26:12.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:12 smithi089 ceph-mon[98732]: pgmap v286: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 150 B/s, 1 objects/s recovering 2023-08-25T02:26:12.964 INFO:teuthology.run_tasks:Running task fs.ready... 2023-08-25T02:26:13.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:26:13.719 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:12 smithi122 ceph-mon[100457]: pgmap v286: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 150 B/s, 1 objects/s recovering 2023-08-25T02:26:13.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:12 smithi121 ceph-mon[100099]: pgmap v286: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 150 B/s, 1 objects/s recovering 2023-08-25T02:26:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 -- 172.21.15.89:0/3711314338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 msgr2=0x7fb3f00ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/3711314338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f00ff3b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400aad0 tx=0x7fb3e400ade0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 -- 172.21.15.89:0/3711314338 shutdown_connections 2023-08-25T02:26:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/3711314338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100b20 0x7fb3f01073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/3711314338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f00ff3b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/3711314338 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100180 0x7fb3f01005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 -- 172.21.15.89:0/3711314338 >> 172.21.15.89:0/3711314338 conn(0x7fb3f00fa4e0 msgr2=0x7fb3f00fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.851+0000 7fb3f61f2700 1 -- 172.21.15.89:0/3711314338 shutdown_connections 2023-08-25T02:26:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 -- 172.21.15.89:0/3711314338 wait complete. 2023-08-25T02:26:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 Processor -- start 2023-08-25T02:26:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 -- start start 2023-08-25T02:26:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100180 0x7fb3f0194980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 0x7fb3f019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f019bfc0 con 0x7fb3f00fef90 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f019c100 con 0x7fb3f0100180 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.852+0000 7fb3f61f2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f019c290 con 0x7fb3f0100b20 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47620/0 (socket says 172.21.15.89:47620) 2023-08-25T02:26:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 -- 172.21.15.89:0/1187691481 learned_addr learned my addr 172.21.15.89:0/1187691481 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3effff700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 0x7fb3f019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3eeffd700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100180 0x7fb3f0194980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 -- 172.21.15.89:0/1187691481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 msgr2=0x7fb3f019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 0x7fb3f019b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 -- 172.21.15.89:0/1187691481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100180 msgr2=0x7fb3f0194980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100180 0x7fb3f0194980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 -- 172.21.15.89:0/1187691481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e400a700 con 0x7fb3f00fef90 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3effff700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 0x7fb3f019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ef7fe700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400af50 tx=0x7fb3e4003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e401e070 con 0x7fb3f00fef90 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3e400ee00 con 0x7fb3f00fef90 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f019c510 con 0x7fb3f00fef90 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4023ad0 con 0x7fb3f00fef90 2023-08-25T02:26:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.853+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f019c8d0 con 0x7fb3f00fef90 2023-08-25T02:26:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.854+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f0105f10 con 0x7fb3f00fef90 2023-08-25T02:26:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.855+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fb3e4023c30 con 0x7fb3f00fef90 2023-08-25T02:26:13.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.856+0000 7fb3ecff9700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 0x7fb3d80a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.856+0000 7fb3eeffd700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 0x7fb3d80a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.856+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb3e40cc860 con 0x7fb3f00fef90 2023-08-25T02:26:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.856+0000 7fb3eeffd700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 0x7fb3d80a2460 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0005950 tx=0x7fb3e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:13.859+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fb3e40966f0 con 0x7fb3f00fef90 2023-08-25T02:26:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:13 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1317352935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-08-25T02:26:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:13 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1317352935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-08-25T02:26:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.051+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3f010a1d0 con 0x7fb3f00fef90 2023-08-25T02:26:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.053+0000 7fb3ecff9700 1 -- 172.21.15.89:0/1187691481 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22576 (secure 0 0 0) 0x7fb3e4099d10 con 0x7fb3f00fef90 2023-08-25T02:26:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:14.055 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:26:04.395929+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:26:14.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 msgr2=0x7fb3d80a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 0x7fb3d80a2460 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0005950 tx=0x7fb3e00058e0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 msgr2=0x7fb3f0194440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400af50 tx=0x7fb3e4003960 comp rx=0 tx=0).stop 2023-08-25T02:26:14.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 shutdown_connections 2023-08-25T02:26:14.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3f0100b20 0x7fb3f019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f00fef90 0x7fb3f0194440 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb3d809ffa0 0x7fb3d80a2460 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 --2- 172.21.15.89:0/1187691481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3f0100180 0x7fb3f0194980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 >> 172.21.15.89:0/1187691481 conn(0x7fb3f00fa4e0 msgr2=0x7fb3f00fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 shutdown_connections 2023-08-25T02:26:14.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.055+0000 7fb3f61f2700 1 -- 172.21.15.89:0/1187691481 wait complete. 2023-08-25T02:26:14.067 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:13 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1317352935' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-08-25T02:26:14.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:26:14.069 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:26:14.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 -- 172.21.15.89:0/2631429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e58100d90 msgr2=0x7f6e581011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2631429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e58100d90 0x7f6e581011b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009990 tx=0x7f6e48009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 -- 172.21.15.89:0/2631429024 shutdown_connections 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2631429024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 0x7f6e58107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2631429024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e58100d90 0x7f6e581011b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2631429024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e580ffac0 0x7f6e580fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 -- 172.21.15.89:0/2631429024 >> 172.21.15.89:0/2631429024 conn(0x7f6e580fa4f0 msgr2=0x7f6e580fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 -- 172.21.15.89:0/2631429024 shutdown_connections 2023-08-25T02:26:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.193+0000 7f6e5e871700 1 -- 172.21.15.89:0/2631429024 wait complete. 2023-08-25T02:26:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 Processor -- start 2023-08-25T02:26:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 -- start start 2023-08-25T02:26:14.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 0x7f6e58196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 0x7f6e5819dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5819e170 con 0x7f6e580ffac0 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5819e2b0 con 0x7f6e58100d90 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5e871700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5819e440 con 0x7f6e58100460 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47634/0 (socket says 172.21.15.89:47634) 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 -- 172.21.15.89:0/2006624425 learned_addr learned my addr 172.21.15.89:0/2006624425 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e577fe700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 0x7f6e58196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 -- 172.21.15.89:0/2006624425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 msgr2=0x7f6e58196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 0x7f6e58196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5ce0e700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 0x7f6e5819dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 -- 172.21.15.89:0/2006624425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 msgr2=0x7f6e5819dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 0x7f6e5819dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 -- 172.21.15.89:0/2006624425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480095f0 con 0x7f6e580ffac0 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e5ce0e700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 0x7f6e5819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.194+0000 7f6e57fff700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6e480095c0 tx=0x7f6e4800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.195+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4801d070 con 0x7f6e580ffac0 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.195+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5819e6c0 con 0x7f6e580ffac0 2023-08-25T02:26:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.195+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4800fb80 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.195+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48017650 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.195+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5819eae0 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.196+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5810aff0 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.197+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6e48022470 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.197+0000 7f6e557fa700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 0x7f6e400a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.198+0000 7f6e577fe700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 0x7f6e400a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.198+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f6e480ccf00 con 0x7f6e580ffac0 2023-08-25T02:26:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.198+0000 7f6e577fe700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 0x7f6e400a2410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00bbd0 tx=0x7f6e4c00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.201+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6e48096d00 con 0x7f6e580ffac0 2023-08-25T02:26:14.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.394+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e5819f060 con 0x7f6e580ffac0 2023-08-25T02:26:14.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.394+0000 7f6e557fa700 1 -- 172.21.15.89:0/2006624425 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e4809a320 con 0x7f6e580ffac0 2023-08-25T02:26:14.397 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:26:14.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.396+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 msgr2=0x7f6e400a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.396+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 0x7f6e400a2410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00bbd0 tx=0x7f6e4c00b5b0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.396+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 msgr2=0x7f6e581965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.396+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6e480095c0 tx=0x7f6e4800f690 comp rx=0 tx=0).stop 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 shutdown_connections 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6e4009ff50 0x7f6e400a2410 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e58100d90 0x7f6e5819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e580ffac0 0x7f6e581965f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 --2- 172.21.15.89:0/2006624425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e58100460 0x7f6e58196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 >> 172.21.15.89:0/2006624425 conn(0x7f6e580fa4f0 msgr2=0x7f6e580fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 shutdown_connections 2023-08-25T02:26:14.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.397+0000 7f6e5e871700 1 -- 172.21.15.89:0/2006624425 wait complete. 2023-08-25T02:26:14.410 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:26:14.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.532+0000 7f50b135d700 1 -- 172.21.15.89:0/72947829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac103d00 msgr2=0x7f50ac1060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.532+0000 7f50b135d700 1 --2- 172.21.15.89:0/72947829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac103d00 0x7f50ac1060f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f509c009990 tx=0x7f509c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 -- 172.21.15.89:0/72947829 shutdown_connections 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 --2- 172.21.15.89:0/72947829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 0x7f50ac108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 --2- 172.21.15.89:0/72947829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac1013d0 0x7f50ac1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 --2- 172.21.15.89:0/72947829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac103d00 0x7f50ac1060f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 -- 172.21.15.89:0/72947829 >> 172.21.15.89:0/72947829 conn(0x7f50ac0fada0 msgr2=0x7f50ac0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 -- 172.21.15.89:0/72947829 shutdown_connections 2023-08-25T02:26:14.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 -- 172.21.15.89:0/72947829 wait complete. 2023-08-25T02:26:14.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 Processor -- start 2023-08-25T02:26:14.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 -- start start 2023-08-25T02:26:14.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.533+0000 7f50b135d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 0x7f50ac0ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 0x7f50ac0ff8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac100060 con 0x7f50ac1013d0 2023-08-25T02:26:14.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac1001d0 con 0x7f50ac106630 2023-08-25T02:26:14.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac1a0620 con 0x7f50ac103d00 2023-08-25T02:26:14.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47648/0 (socket says 172.21.15.89:47648) 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 -- 172.21.15.89:0/2437269181 learned_addr learned my addr 172.21.15.89:0/2437269181 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aa7fc700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 0x7f50ac0ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50ab7fe700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 0x7f50ac0ff8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 -- 172.21.15.89:0/2437269181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 msgr2=0x7f50ac0ff380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 0x7f50ac0ff380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 -- 172.21.15.89:0/2437269181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 msgr2=0x7f50ac0ff8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 0x7f50ac0ff8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 -- 172.21.15.89:0/2437269181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c0095f0 con 0x7f50ac1013d0 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aa7fc700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 0x7f50ac0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50aaffd700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f509400d950 tx=0x7f509400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094009b40 con 0x7f50ac1013d0 2023-08-25T02:26:14.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac1a0900 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5094009ca0 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509400f620 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.534+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac1a0e60 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.535+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac0fc920 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.536+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5094010460 con 0x7f50ac1013d0 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.537+0000 7f50a3fff700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 0x7f50980a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.537+0000 7f50aa7fc700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 0x7f50980a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.537+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f50940cb5e0 con 0x7f50ac1013d0 2023-08-25T02:26:14.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.537+0000 7f50aa7fc700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 0x7f50980a2410 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f509c006010 tx=0x7f509c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.541+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5094095350 con 0x7f50ac1013d0 2023-08-25T02:26:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.734+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50ac10c3f0 con 0x7f50ac1013d0 2023-08-25T02:26:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.734+0000 7f50a3fff700 1 -- 172.21.15.89:0/2437269181 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5094098970 con 0x7f50ac1013d0 2023-08-25T02:26:14.736 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:26:14.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 msgr2=0x7f50980a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 0x7f50980a2410 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f509c006010 tx=0x7f509c00b540 comp rx=0 tx=0).stop 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 msgr2=0x7f50ac100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f509400d950 tx=0x7f509400dd10 comp rx=0 tx=0).stop 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 shutdown_connections 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f509809ff50 0x7f50980a2410 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50ac106630 0x7f50ac0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50ac1013d0 0x7f50ac100d40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 --2- 172.21.15.89:0/2437269181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50ac103d00 0x7f50ac0ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.736+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 >> 172.21.15.89:0/2437269181 conn(0x7f50ac0fada0 msgr2=0x7f50ac102040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.737+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 shutdown_connections 2023-08-25T02:26:14.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.737+0000 7f50b135d700 1 -- 172.21.15.89:0/2437269181 wait complete. 2023-08-25T02:26:14.749 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:14 smithi089 ceph-mon[98732]: pgmap v287: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 121 B/s, 0 objects/s recovering 2023-08-25T02:26:14.749 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1187691481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:14.749 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:14 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2006624425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:14.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:26:14.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 -- 172.21.15.89:0/1817553865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b81013b0 msgr2=0x7f33b8107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 --2- 172.21.15.89:0/1817553865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b81013b0 0x7f33b8107c60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f33b000b0d0 tx=0x7f33b000b3e0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 -- 172.21.15.89:0/1817553865 shutdown_connections 2023-08-25T02:26:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 --2- 172.21.15.89:0/1817553865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b81013b0 0x7f33b8107c60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 --2- 172.21.15.89:0/1817553865 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b80ff820 0x7f33b80ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 --2- 172.21.15.89:0/1817553865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b8100a10 0x7f33b8100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 -- 172.21.15.89:0/1817553865 >> 172.21.15.89:0/1817553865 conn(0x7f33b80fadb0 msgr2=0x7f33b80fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 -- 172.21.15.89:0/1817553865 shutdown_connections 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.873+0000 7f33bc95a700 1 -- 172.21.15.89:0/1817553865 wait complete. 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 Processor -- start 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 -- start start 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b8100a10 0x7f33b8196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b81013b0 0x7f33b819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b819e1c0 con 0x7f33b80ff820 2023-08-25T02:26:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b819e300 con 0x7f33b81013b0 2023-08-25T02:26:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33bc95a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b819e490 con 0x7f33b8100a10 2023-08-25T02:26:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47662/0 (socket says 172.21.15.89:47662) 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 -- 172.21.15.89:0/936459846 learned_addr learned my addr 172.21.15.89:0/936459846 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b5d9b700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b8100a10 0x7f33b8196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b6d9d700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b81013b0 0x7f33b819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 -- 172.21.15.89:0/936459846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b8100a10 msgr2=0x7f33b8196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b8100a10 0x7f33b8196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 -- 172.21.15.89:0/936459846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b81013b0 msgr2=0x7f33b819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b81013b0 0x7f33b819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 -- 172.21.15.89:0/936459846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0009d00 con 0x7f33b80ff820 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33b659c700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c970 tx=0x7f33ac00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.874+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac004220 con 0x7f33b80ff820 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.875+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33ac007d60 con 0x7f33b80ff820 2023-08-25T02:26:14.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.875+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b819e770 con 0x7f33b80ff820 2023-08-25T02:26:14.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.875+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac003d00 con 0x7f33b80ff820 2023-08-25T02:26:14.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.875+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b819ecd0 con 0x7f33b80ff820 2023-08-25T02:26:14.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.875+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b810b040 con 0x7f33b80ff820 2023-08-25T02:26:14.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.877+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f33ac004380 con 0x7f33b80ff820 2023-08-25T02:26:14.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.877+0000 7f33a77fe700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 0x7f33a00a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:14.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.877+0000 7f33b5d9b700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 0x7f33a00a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:14.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.877+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f33ac0cb150 con 0x7f33b80ff820 2023-08-25T02:26:14.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.877+0000 7f33b5d9b700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 0x7f33a00a2490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f33a8005ea0 tx=0x7f33a8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:14.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:14.880+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f33ac094f40 con 0x7f33b80ff820 2023-08-25T02:26:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:14 smithi121 ceph-mon[100099]: pgmap v287: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 121 B/s, 0 objects/s recovering 2023-08-25T02:26:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:14 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1187691481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:14 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2006624425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:14 smithi122 ceph-mon[100457]: pgmap v287: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 121 B/s, 0 objects/s recovering 2023-08-25T02:26:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:14 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1187691481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:14 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2006624425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.070+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33b819ef50 con 0x7f33b80ff820 2023-08-25T02:26:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.070+0000 7f33a77fe700 1 -- 172.21.15.89:0/936459846 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f33ac01d020 con 0x7f33b80ff820 2023-08-25T02:26:15.074 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:26:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 msgr2=0x7f33a00a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 0x7f33a00a2490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f33a8005ea0 tx=0x7f33a8005e30 comp rx=0 tx=0).stop 2023-08-25T02:26:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 msgr2=0x7f33b8196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c970 tx=0x7f33ac00cd30 comp rx=0 tx=0).stop 2023-08-25T02:26:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 shutdown_connections 2023-08-25T02:26:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33a009ffd0 0x7f33a00a2490 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33b81013b0 0x7f33b819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33b80ff820 0x7f33b8196640 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 --2- 172.21.15.89:0/936459846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33b8100a10 0x7f33b8196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 >> 172.21.15.89:0/936459846 conn(0x7f33b80fadb0 msgr2=0x7f33b8068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 shutdown_connections 2023-08-25T02:26:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.073+0000 7f33bc95a700 1 -- 172.21.15.89:0/936459846 wait complete. 2023-08-25T02:26:15.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:26:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- 172.21.15.89:0/1321763623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c8100920 msgr2=0x7f37c8100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- 172.21.15.89:0/1321763623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c8100920 0x7f37c8100d40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f37c40099c0 tx=0x7f37c4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- 172.21.15.89:0/1321763623 shutdown_connections 2023-08-25T02:26:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- 172.21.15.89:0/1321763623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c81024b0 0x7f37c8108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- 172.21.15.89:0/1321763623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c8100920 0x7f37c8100d40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- 172.21.15.89:0/1321763623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- 172.21.15.89:0/1321763623 >> 172.21.15.89:0/1321763623 conn(0x7f37c80fbe70 msgr2=0x7f37c80fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- 172.21.15.89:0/1321763623 shutdown_connections 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- 172.21.15.89:0/1321763623 wait complete. 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 Processor -- start 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- start start 2023-08-25T02:26:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c8100920 0x7f37c8196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c819e0a0 con 0x7f37c81024b0 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c819e1e0 con 0x7f37c8101b10 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.208+0000 7f37d0460700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c819e370 con 0x7f37c8100920 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47680/0 (socket says 172.21.15.89:47680) 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 -- 172.21.15.89:0/1527674313 learned_addr learned my addr 172.21.15.89:0/1527674313 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37cd9fb700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce1fc700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c8100920 0x7f37c8196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 -- 172.21.15.89:0/1527674313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c8100920 msgr2=0x7f37c8196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c8100920 0x7f37c8196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 -- 172.21.15.89:0/1527674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 msgr2=0x7f37c8196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 -- 172.21.15.89:0/1527674313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40095f0 con 0x7f37c81024b0 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37cd9fb700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37ce9fd700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007ee0 tx=0x7f37c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0017070 con 0x7f37c81024b0 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37c000f040 con 0x7f37c81024b0 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c819e650 con 0x7f37c81024b0 2023-08-25T02:26:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0005740 con 0x7f37c81024b0 2023-08-25T02:26:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.209+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c819eb60 con 0x7f37c81024b0 2023-08-25T02:26:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.210+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c810c130 con 0x7f37c81024b0 2023-08-25T02:26:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.211+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f37c00058a0 con 0x7f37c81024b0 2023-08-25T02:26:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.212+0000 7f37bf7fe700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 0x7f37b40a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.212+0000 7f37ce1fc700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 0x7f37b40a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.212+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f37c00cb0c0 con 0x7f37c81024b0 2023-08-25T02:26:15.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.212+0000 7f37ce1fc700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 0x7f37b40a24e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f37c400b580 tx=0x7f37c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.215+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f37c0094e30 con 0x7f37c81024b0 2023-08-25T02:26:15.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.449+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c81077b0 con 0x7f37c81024b0 2023-08-25T02:26:15.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.449+0000 7f37bf7fe700 1 -- 172.21.15.89:0/1527674313 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7f37c0098450 con 0x7f37c81024b0 2023-08-25T02:26:15.451 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:15.451 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:26:15.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 msgr2=0x7f37b40a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 0x7f37b40a24e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f37c400b580 tx=0x7f37c4005fb0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 msgr2=0x7f37c819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007ee0 tx=0x7f37c000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 shutdown_connections 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37c81024b0 0x7f37c819dad0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c8100920 0x7f37c8196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f37b40a0020 0x7f37b40a24e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 --2- 172.21.15.89:0/1527674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37c8101b10 0x7f37c8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.451+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 >> 172.21.15.89:0/1527674313 conn(0x7f37c80fbe70 msgr2=0x7f37c80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.452+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 shutdown_connections 2023-08-25T02:26:15.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.452+0000 7f37d0460700 1 -- 172.21.15.89:0/1527674313 wait complete. 2023-08-25T02:26:15.455 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:26:15.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:26:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 -- 172.21.15.89:0/95459260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 msgr2=0x7f0370108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 --2- 172.21.15.89:0/95459260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370108d70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f036800b0d0 tx=0x7f036800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 -- 172.21.15.89:0/95459260 shutdown_connections 2023-08-25T02:26:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 --2- 172.21.15.89:0/95459260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370108d70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 --2- 172.21.15.89:0/95459260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370100930 0x7f0370100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 --2- 172.21.15.89:0/95459260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370101b20 0x7f0370101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 -- 172.21.15.89:0/95459260 >> 172.21.15.89:0/95459260 conn(0x7f03700fbe60 msgr2=0x7f03700fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 -- 172.21.15.89:0/95459260 shutdown_connections 2023-08-25T02:26:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 -- 172.21.15.89:0/95459260 wait complete. 2023-08-25T02:26:15.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.586+0000 7f0375095700 1 Processor -- start 2023-08-25T02:26:15.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 -- start start 2023-08-25T02:26:15.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370100930 0x7f03700746d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370101b20 0x7f0370072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03700739f0 con 0x7f03701024c0 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0370073b60 con 0x7f0370100930 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0375095700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701a0660 con 0x7f0370101b20 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47694/0 (socket says 172.21.15.89:47694) 2023-08-25T02:26:15.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 -- 172.21.15.89:0/2905000413 learned_addr learned my addr 172.21.15.89:0/2905000413 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036e59c700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370101b20 0x7f0370072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036ed9d700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370100930 0x7f03700746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 -- 172.21.15.89:0/2905000413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370101b20 msgr2=0x7f0370072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370101b20 0x7f0370072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 -- 172.21.15.89:0/2905000413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370100930 msgr2=0x7f03700746d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370100930 0x7f03700746d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 -- 172.21.15.89:0/2905000413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368009d00 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f036f59e700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0368007a00 tx=0x7f03680094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.587+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036800e040 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.588+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0368003e10 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.588+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368012a90 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.588+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03701a08e0 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.588+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03701a0cb0 con 0x7f03701024c0 2023-08-25T02:26:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.588+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0370107d30 con 0x7f03701024c0 2023-08-25T02:26:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.590+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f03680056c0 con 0x7f03701024c0 2023-08-25T02:26:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.591+0000 7f0357fff700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 0x7f03580a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.591+0000 7f036ed9d700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 0x7f03580a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.591+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f03680cd0d0 con 0x7f03701024c0 2023-08-25T02:26:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.591+0000 7f036ed9d700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 0x7f03580a2460 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0364005950 tx=0x7f03640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.594+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0368096ed0 con 0x7f03701024c0 2023-08-25T02:26:15.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.789+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f037002ce80 con 0x7f03701024c0 2023-08-25T02:26:15.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.791+0000 7f0357fff700 1 -- 172.21.15.89:0/2905000413 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22576 (secure 0 0 0) 0x7f036809a4f0 con 0x7f03701024c0 2023-08-25T02:26:15.792 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:15.793 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:26:04.395929+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:26:15.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 msgr2=0x7f03580a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 0x7f03580a2460 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0364005950 tx=0x7f03640058e0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 msgr2=0x7f0370073250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0368007a00 tx=0x7f03680094c0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 shutdown_connections 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f035809ffa0 0x7f03580a2460 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0370100930 0x7f03700746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03701024c0 0x7f0370073250 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 --2- 172.21.15.89:0/2905000413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370101b20 0x7f0370072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 >> 172.21.15.89:0/2905000413 conn(0x7f03700fbe60 msgr2=0x7f0370104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 shutdown_connections 2023-08-25T02:26:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.793+0000 7f0375095700 1 -- 172.21.15.89:0/2905000413 wait complete. 2023-08-25T02:26:15.805 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:15 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2437269181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.805 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:15 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/936459846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.806 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:15 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1527674313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:15.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:26:15.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:26:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 -- 172.21.15.89:0/2546812627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100460 msgr2=0x7fbc34107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2546812627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100460 0x7fbc34107d00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00b0a0 tx=0x7fbc2c00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 -- 172.21.15.89:0/2546812627 shutdown_connections 2023-08-25T02:26:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2546812627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc34100e00 0x7fbc34101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2546812627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100460 0x7fbc34107d00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2546812627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc340ffaa0 0x7fbc340fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 -- 172.21.15.89:0/2546812627 >> 172.21.15.89:0/2546812627 conn(0x7fbc340fa4f0 msgr2=0x7fbc340fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 -- 172.21.15.89:0/2546812627 shutdown_connections 2023-08-25T02:26:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.930+0000 7fbc3c503700 1 -- 172.21.15.89:0/2546812627 wait complete. 2023-08-25T02:26:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 Processor -- start 2023-08-25T02:26:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 -- start start 2023-08-25T02:26:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc340ffaa0 0x7fbc34103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 0x7fbc34101e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34102b70 con 0x7fbc34100e00 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34102ce0 con 0x7fbc340ffaa0 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3c503700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341a07e0 con 0x7fbc34100460 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47708/0 (socket says 172.21.15.89:47708) 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 -- 172.21.15.89:0/2278656345 learned_addr learned my addr 172.21.15.89:0/2278656345 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3a29f700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc340ffaa0 0x7fbc34103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc39a9e700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 0x7fbc34101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 -- 172.21.15.89:0/2278656345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 msgr2=0x7fbc34101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 0x7fbc34101e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 -- 172.21.15.89:0/2278656345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc340ffaa0 msgr2=0x7fbc34103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc340ffaa0 0x7fbc34103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc3aaa0700 1 -- 172.21.15.89:0/2278656345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c009d00 con 0x7fbc34100e00 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.931+0000 7fbc39a9e700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 0x7fbc34101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc3aaa0700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0062a0 tx=0x7fbc2c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c00e040 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc2c00cec0 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc341a0a60 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c004020 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.932+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341a0ee0 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.933+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3410b020 con 0x7fbc34100e00 2023-08-25T02:26:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.934+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fbc2c01d690 con 0x7fbc34100e00 2023-08-25T02:26:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.934+0000 7fbc2b7fe700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 0x7fbc200a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.934+0000 7fbc3a29f700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 0x7fbc200a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.934+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbc2c0cca20 con 0x7fbc34100e00 2023-08-25T02:26:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.935+0000 7fbc3a29f700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 0x7fbc200a2410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc34067820 tx=0x7fbc2400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:15.939+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fbc2c096820 con 0x7fbc34100e00 2023-08-25T02:26:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:15 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2437269181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:15 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/936459846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:15 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1527674313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:15.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:15 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2437269181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:15 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/936459846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:15.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:15 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1527674313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:16.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.132+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc341a10d0 con 0x7fbc34100e00 2023-08-25T02:26:16.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.132+0000 7fbc2b7fe700 1 -- 172.21.15.89:0/2278656345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc2c099e40 con 0x7fbc34100e00 2023-08-25T02:26:16.133 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:26:16.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 msgr2=0x7fbc200a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 0x7fbc200a2410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc34067820 tx=0x7fbc2400b410 comp rx=0 tx=0).stop 2023-08-25T02:26:16.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 msgr2=0x7fbc341023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0062a0 tx=0x7fbc2c004d00 comp rx=0 tx=0).stop 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 shutdown_connections 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbc2009ff50 0x7fbc200a2410 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc340ffaa0 0x7fbc34103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc34100e00 0x7fbc341023d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 --2- 172.21.15.89:0/2278656345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc34100460 0x7fbc34101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 >> 172.21.15.89:0/2278656345 conn(0x7fbc340fa4f0 msgr2=0x7fbc340fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 shutdown_connections 2023-08-25T02:26:16.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.134+0000 7fbc3c503700 1 -- 172.21.15.89:0/2278656345 wait complete. 2023-08-25T02:26:16.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 -- 172.21.15.89:0/945706988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 msgr2=0x7f33e4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 --2- 172.21.15.89:0/945706988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e4100d60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f33d40099c0 tx=0x7f33d4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 -- 172.21.15.89:0/945706988 shutdown_connections 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 --2- 172.21.15.89:0/945706988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e4108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 --2- 172.21.15.89:0/945706988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e4100d60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 --2- 172.21.15.89:0/945706988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33e4101b30 0x7f33e4101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.273+0000 7f33eba5e700 1 -- 172.21.15.89:0/945706988 >> 172.21.15.89:0/945706988 conn(0x7f33e40fbe90 msgr2=0x7f33e40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- 172.21.15.89:0/945706988 shutdown_connections 2023-08-25T02:26:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- 172.21.15.89:0/945706988 wait complete. 2023-08-25T02:26:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 Processor -- start 2023-08-25T02:26:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- start start 2023-08-25T02:26:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33e4101b30 0x7f33e4196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e419e160 con 0x7f33e4100940 2023-08-25T02:26:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e419e2a0 con 0x7f33e4101b30 2023-08-25T02:26:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33eba5e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e419e430 con 0x7f33e41024d0 2023-08-25T02:26:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47718/0 (socket says 172.21.15.89:47718) 2023-08-25T02:26:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 -- 172.21.15.89:0/1795731185 learned_addr learned my addr 172.21.15.89:0/1795731185 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e9ffb700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 -- 172.21.15.89:0/1795731185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 msgr2=0x7f33e419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e419db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.274+0000 7f33e97fa700 1 -- 172.21.15.89:0/1795731185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33e4101b30 msgr2=0x7f33e4196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33e97fa700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33e4101b30 0x7f33e4196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33e97fa700 1 -- 172.21.15.89:0/1795731185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40095f0 con 0x7f33e4100940 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33e9ffb700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33e97fa700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009990 tx=0x7f33d4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d401d070 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33d4004200 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4022e30 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e419e6b0 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.275+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e419eb60 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.276+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e4107ee0 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.277+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f33d4003d40 con 0x7f33e4100940 2023-08-25T02:26:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.277+0000 7f33da7fc700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 0x7f33d00a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.277+0000 7f33e8ff9700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 0x7f33d00a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.277+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f33d40cc980 con 0x7f33e4100940 2023-08-25T02:26:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.278+0000 7f33e8ff9700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 0x7f33d00a2490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005d90 tx=0x7f33e0005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.281+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f33d4096780 con 0x7f33e4100940 2023-08-25T02:26:16.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.473+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33e410c2e0 con 0x7f33e4100940 2023-08-25T02:26:16.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.473+0000 7f33da7fc700 1 -- 172.21.15.89:0/1795731185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33d40270a0 con 0x7f33e4100940 2023-08-25T02:26:16.476 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:26:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 msgr2=0x7f33d00a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 0x7f33d00a2490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005d90 tx=0x7f33e0005d00 comp rx=0 tx=0).stop 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 msgr2=0x7f33e41965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009990 tx=0x7f33d4005e80 comp rx=0 tx=0).stop 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 shutdown_connections 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33e41024d0 0x7f33e419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4100940 0x7f33e41965e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f33d009ffd0 0x7f33d00a2490 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 --2- 172.21.15.89:0/1795731185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33e4101b30 0x7f33e4196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 >> 172.21.15.89:0/1795731185 conn(0x7f33e40fbe90 msgr2=0x7f33e4105020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 shutdown_connections 2023-08-25T02:26:16.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.476+0000 7f33eba5e700 1 -- 172.21.15.89:0/1795731185 wait complete. 2023-08-25T02:26:16.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:26:16.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 -- 172.21.15.89:0/1873660823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0106650 msgr2=0x7f6fb0108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/1873660823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0106650 0x7f6fb0108a60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800b0a0 tx=0x7f6fa800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 -- 172.21.15.89:0/1873660823 shutdown_connections 2023-08-25T02:26:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/1873660823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fb01013f0 0x7f6fb01037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/1873660823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0106650 0x7f6fb0108a60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/1873660823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb0103d20 0x7f6fb0106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 -- 172.21.15.89:0/1873660823 >> 172.21.15.89:0/1873660823 conn(0x7f6fb00fad80 msgr2=0x7f6fb00fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 -- 172.21.15.89:0/1873660823 shutdown_connections 2023-08-25T02:26:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.612+0000 7f6fb7a96700 1 -- 172.21.15.89:0/1873660823 wait complete. 2023-08-25T02:26:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 Processor -- start 2023-08-25T02:26:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 -- start start 2023-08-25T02:26:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb01013f0 0x7f6fb0196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fb0106650 0x7f6fb019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb019e280 con 0x7f6fb0103d20 2023-08-25T02:26:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb019e3c0 con 0x7f6fb0106650 2023-08-25T02:26:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb019e550 con 0x7f6fb01013f0 2023-08-25T02:26:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47734/0 (socket says 172.21.15.89:47734) 2023-08-25T02:26:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 -- 172.21.15.89:0/2772557944 learned_addr learned my addr 172.21.15.89:0/2772557944 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 -- 172.21.15.89:0/2772557944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb01013f0 msgr2=0x7f6fb0196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb01013f0 0x7f6fb0196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 -- 172.21.15.89:0/2772557944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fb0106650 msgr2=0x7f6fb019dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fb0106650 0x7f6fb019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 -- 172.21.15.89:0/2772557944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8009d00 con 0x7f6fb0103d20 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb5031700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00d950 tx=0x7f6fac00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac009b40 con 0x7f6fb0103d20 2023-08-25T02:26:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac009ca0 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb019e830 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.613+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00f620 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.614+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb019ed40 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.614+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb00fc900 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.615+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f6fac010460 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.616+0000 7f6fa6ffd700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 0x7f6f9c0a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.616+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f6fac0cb5e0 con 0x7f6fb0103d20 2023-08-25T02:26:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.616+0000 7f6fb5832700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 0x7f6f9c0a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.616+0000 7f6fb5832700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 0x7f6f9c0a2410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005950 tx=0x7f6fa00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.619+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6fac095350 con 0x7f6fb0103d20 2023-08-25T02:26:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.812+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fb002cc20 con 0x7f6fb0103d20 2023-08-25T02:26:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.813+0000 7f6fa6ffd700 1 -- 172.21.15.89:0/2772557944 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6fac020020 con 0x7f6fb0103d20 2023-08-25T02:26:16.815 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:26:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 msgr2=0x7f6f9c0a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 0x7f6f9c0a2410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005950 tx=0x7f6fa00058e0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 msgr2=0x7f6fb0196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00d950 tx=0x7f6fac00dd10 comp rx=0 tx=0).stop 2023-08-25T02:26:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 shutdown_connections 2023-08-25T02:26:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f6f9c09ff50 0x7f6f9c0a2410 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fb0106650 0x7f6fb019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fb01013f0 0x7f6fb0196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 --2- 172.21.15.89:0/2772557944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb0103d20 0x7f6fb0196c40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 >> 172.21.15.89:0/2772557944 conn(0x7f6fb00fad80 msgr2=0x7f6fb0102060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 shutdown_connections 2023-08-25T02:26:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.815+0000 7f6fb7a96700 1 -- 172.21.15.89:0/2772557944 wait complete. 2023-08-25T02:26:16.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:16 smithi089 ceph-mon[98732]: pgmap v288: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 0 objects/s recovering 2023-08-25T02:26:16.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:16 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2905000413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:16.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:16 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2278656345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:16.828 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:16 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1795731185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:16.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:26:16.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- 172.21.15.89:0/3139668719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 msgr2=0x7fba381037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- 172.21.15.89:0/3139668719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba381037d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009990 tx=0x7fba2c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- 172.21.15.89:0/3139668719 shutdown_connections 2023-08-25T02:26:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- 172.21.15.89:0/3139668719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38106640 0x7fba38108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- 172.21.15.89:0/3139668719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba381037d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- 172.21.15.89:0/3139668719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38103d10 0x7fba38106100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- 172.21.15.89:0/3139668719 >> 172.21.15.89:0/3139668719 conn(0x7fba380fad50 msgr2=0x7fba380fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- 172.21.15.89:0/3139668719 shutdown_connections 2023-08-25T02:26:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- 172.21.15.89:0/3139668719 wait complete. 2023-08-25T02:26:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 Processor -- start 2023-08-25T02:26:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- start start 2023-08-25T02:26:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 0x7fba380ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38106640 0x7fba380ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.951+0000 7fba3f233700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38100070 con 0x7fba381013e0 2023-08-25T02:26:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3f233700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381001e0 con 0x7fba38103d10 2023-08-25T02:26:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3f233700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381a0610 con 0x7fba38106640 2023-08-25T02:26:16.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47746/0 (socket says 172.21.15.89:47746) 2023-08-25T02:26:16.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 -- 172.21.15.89:0/1538510953 learned_addr learned my addr 172.21.15.89:0/1538510953 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:16.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba37fff700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 0x7fba380ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3d7d0700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38106640 0x7fba380ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 -- 172.21.15.89:0/1538510953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38106640 msgr2=0x7fba380ff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38106640 0x7fba380ff8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 -- 172.21.15.89:0/1538510953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 msgr2=0x7fba380ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 0x7fba380ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 -- 172.21.15.89:0/1538510953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c0095f0 con 0x7fba381013e0 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba37fff700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 0x7fba380ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3cfcf700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0059d0 tx=0x7fba2c00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01d070 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba381a0890 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba2c00fd60 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c017650 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.952+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba381a0cc0 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.953+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba380fc8d0 con 0x7fba381013e0 2023-08-25T02:26:16.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.954+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fba2c0177b0 con 0x7fba381013e0 2023-08-25T02:26:16.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.955+0000 7fba35ffb700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 0x7fba280a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:16.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.955+0000 7fba37fff700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 0x7fba280a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:16.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.955+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fba2c0cc7a0 con 0x7fba381013e0 2023-08-25T02:26:16.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.955+0000 7fba37fff700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 0x7fba280a2460 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fba24009cb0 tx=0x7fba24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:16.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:16.959+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fba2c05ce00 con 0x7fba381013e0 2023-08-25T02:26:16.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:16 smithi121 ceph-mon[100099]: pgmap v288: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 0 objects/s recovering 2023-08-25T02:26:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:16 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2905000413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:16 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2278656345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:16.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:16 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1795731185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:16 smithi122 ceph-mon[100457]: pgmap v288: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 0 objects/s recovering 2023-08-25T02:26:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:16 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2905000413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:16.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:16 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2278656345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:26:16.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:16 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1795731185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:26:17.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.190+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba3810c0e0 con 0x7fba381013e0 2023-08-25T02:26:17.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.191+0000 7fba35ffb700 1 -- 172.21.15.89:0/1538510953 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11286 (secure 0 0 0) 0x7fba2c0965a0 con 0x7fba381013e0 2023-08-25T02:26:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:17.192 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:25:09.635801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:26:17.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 msgr2=0x7fba280a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 0x7fba280a2460 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fba24009cb0 tx=0x7fba24009260 comp rx=0 tx=0).stop 2023-08-25T02:26:17.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 msgr2=0x7fba38100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0059d0 tx=0x7fba2c00f510 comp rx=0 tx=0).stop 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 shutdown_connections 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba38106640 0x7fba380ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba381013e0 0x7fba38100d50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fba2809ffa0 0x7fba280a2460 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 --2- 172.21.15.89:0/1538510953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba38103d10 0x7fba380ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 >> 172.21.15.89:0/1538510953 conn(0x7fba380fad50 msgr2=0x7fba381020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 shutdown_connections 2023-08-25T02:26:17.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.193+0000 7fba3f233700 1 -- 172.21.15.89:0/1538510953 wait complete. 2023-08-25T02:26:17.196 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-08-25T02:26:17.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:26:17.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.326+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/3344254610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf8101390 msgr2=0x7f8bf8107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.326+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/3344254610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf8101390 0x7f8bf8107c40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000b0d0 tx=0x7f8bf000b3e0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.326+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/3344254610 shutdown_connections 2023-08-25T02:26:17.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.326+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/3344254610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf8101390 0x7f8bf8107c40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/3344254610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf80ff800 0x7f8bf80ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/3344254610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf81009f0 0x7f8bf8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/3344254610 >> 172.21.15.89:0/3344254610 conn(0x7f8bf80fad50 msgr2=0x7f8bf80fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/3344254610 shutdown_connections 2023-08-25T02:26:17.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/3344254610 wait complete. 2023-08-25T02:26:17.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.327+0000 7f8bfd1a4700 1 Processor -- start 2023-08-25T02:26:17.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 -- start start 2023-08-25T02:26:17.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf80ff800 0x7f8bf8195630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 0x7f8bf819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf819d1b0 con 0x7f8bf81009f0 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf819d2f0 con 0x7f8bf80ff800 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bfd1a4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf819d480 con 0x7f8bf8101390 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47760/0 (socket says 172.21.15.89:47760) 2023-08-25T02:26:17.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 -- 172.21.15.89:0/147862638 learned_addr learned my addr 172.21.15.89:0/147862638 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf6d9d700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf80ff800 0x7f8bf8195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf759e700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 0x7f8bf819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 -- 172.21.15.89:0/147862638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 msgr2=0x7f8bf819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 0x7f8bf819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 -- 172.21.15.89:0/147862638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf80ff800 msgr2=0x7f8bf8195630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf80ff800 0x7f8bf8195630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 -- 172.21.15.89:0/147862638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf0009d00 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf759e700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 0x7f8bf819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.328+0000 7f8bf659c700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8be800d950 tx=0x7f8be800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.329+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8009940 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.329+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8be8010460 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.329+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf8067ae0 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.329+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be800f620 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.329+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf8067fe0 con 0x7f8bf81009f0 2023-08-25T02:26:17.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.330+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf8067710 con 0x7f8bf81009f0 2023-08-25T02:26:17.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.331+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f8be80108d0 con 0x7f8bf81009f0 2023-08-25T02:26:17.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.332+0000 7f8bdffff700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 0x7f8be00a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.332+0000 7f8bf6d9d700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 0x7f8be00a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.332+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f8be80cb070 con 0x7f8bf81009f0 2023-08-25T02:26:17.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.332+0000 7f8bf6d9d700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 0x7f8be00a2460 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8100850 tx=0x7f8bec009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.335+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8be8094de0 con 0x7f8bf81009f0 2023-08-25T02:26:17.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.529+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bf810b300 con 0x7f8bf81009f0 2023-08-25T02:26:17.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.531+0000 7f8bdffff700 1 -- 172.21.15.89:0/147862638 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22576 (secure 0 0 0) 0x7f8be8098400 con 0x7f8bf81009f0 2023-08-25T02:26:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:17.533 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:26:04.395929+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:26:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 msgr2=0x7f8be00a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 0x7f8be00a2460 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8100850 tx=0x7f8bec009190 comp rx=0 tx=0).stop 2023-08-25T02:26:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 msgr2=0x7f8bf8195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8be800d950 tx=0x7f8be800dc60 comp rx=0 tx=0).stop 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 shutdown_connections 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f8be009ffa0 0x7f8be00a2460 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf80ff800 0x7f8bf8195630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bf8101390 0x7f8bf819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 --2- 172.21.15.89:0/147862638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bf81009f0 0x7f8bf8195b70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 >> 172.21.15.89:0/147862638 conn(0x7f8bf80fad50 msgr2=0x7f8bf8103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 shutdown_connections 2023-08-25T02:26:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.533+0000 7f8bfd1a4700 1 -- 172.21.15.89:0/147862638 wait complete. 2023-08-25T02:26:17.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-08-25T02:25:04.024043+0000', 'modified': '2023-08-25T02:25:09.635801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24475, 'mds_2': 24470}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.121:6835/217468698', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 217468698}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 217468698}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6835/3100558972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 3100558972}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 3100558972}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6837/2359799361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2359799361}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2359799361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.121:6837/2048790446', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6836', 'nonce': 2048790446}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 2048790446}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.122:6839/3754052688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 3754052688}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 3754052688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6839/1864669733', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6838', 'nonce': 1864669733}, {'type': 'v1', 'addr': '172.21.15.121:6839', 'nonce': 1864669733}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-08-25T02:26:17.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-08-25T02:26:17.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-08-25T02:26:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 -- 172.21.15.89:0/1996427029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc103d20 msgr2=0x7fb5fc106110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 --2- 172.21.15.89:0/1996427029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc103d20 0x7fb5fc106110 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009990 tx=0x7fb5f0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 -- 172.21.15.89:0/1996427029 shutdown_connections 2023-08-25T02:26:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 --2- 172.21.15.89:0/1996427029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 --2- 172.21.15.89:0/1996427029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 0x7fb5fc108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 --2- 172.21.15.89:0/1996427029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc103d20 0x7fb5fc106110 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 -- 172.21.15.89:0/1996427029 >> 172.21.15.89:0/1996427029 conn(0x7fb5fc0fad80 msgr2=0x7fb5fc0fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 -- 172.21.15.89:0/1996427029 shutdown_connections 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.671+0000 7fb604452700 1 -- 172.21.15.89:0/1996427029 wait complete. 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 Processor -- start 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 -- start start 2023-08-25T02:26:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5fc103d20 0x7fb5fc196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 0x7fb5fc19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc19e190 con 0x7fb5fc1013f0 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc19e2d0 con 0x7fb5fc103d20 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb604452700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc19e460 con 0x7fb5fc106650 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47772/0 (socket says 172.21.15.89:47772) 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 -- 172.21.15.89:0/723729774 learned_addr learned my addr 172.21.15.89:0/723729774 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6029ef700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 0x7fb5fc19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 -- 172.21.15.89:0/723729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 msgr2=0x7fb5fc19dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 0x7fb5fc19dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 -- 172.21.15.89:0/723729774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5fc103d20 msgr2=0x7fb5fc196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5fc103d20 0x7fb5fc196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.672+0000 7fb6021ee700 1 -- 172.21.15.89:0/723729774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00095f0 con 0x7fb5fc1013f0 2023-08-25T02:26:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb6021ee700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800c920 tx=0x7fb5f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb5ef7fe700 1 -- 172.21.15.89:0/723729774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8007b00 con 0x7fb5fc1013f0 2023-08-25T02:26:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb5ef7fe700 1 -- 172.21.15.89:0/723729774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5f8004cf0 con 0x7fb5fc1013f0 2023-08-25T02:26:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc19e740 con 0x7fb5fc1013f0 2023-08-25T02:26:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb5ef7fe700 1 -- 172.21.15.89:0/723729774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8005610 con 0x7fb5fc1013f0 2023-08-25T02:26:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.673+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc19eca0 con 0x7fb5fc1013f0 2023-08-25T02:26:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.675+0000 7fb5ef7fe700 1 -- 172.21.15.89:0/723729774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fb5f8004750 con 0x7fb5fc1013f0 2023-08-25T02:26:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.675+0000 7fb5ef7fe700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 0x7fb5e80a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.675+0000 7fb5ef7fe700 1 -- 172.21.15.89:0/723729774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb5f80cb1b0 con 0x7fb5fc1013f0 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.675+0000 7fb6019ed700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 0x7fb5e80a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.676+0000 7fb6019ed700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 0x7fb5e80a2490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00095c0 tx=0x7fb5f0009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.676+0000 7fb6105f6b80 1 Processor -- start 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 -- start start 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748c980 0x557c9748cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 0x557c9748d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c973f4760 con 0x557c9748dd00 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c973e5730 con 0x557c9748c980 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c973e3050 con 0x557c9748d2e0 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47784/0 (socket says 172.21.15.89:47784) 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 -- 172.21.15.89:0/963019376 learned_addr learned my addr 172.21.15.89:0/963019376 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6021ee700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748c980 0x557c9748cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6019ed700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 0x557c9748d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 -- 172.21.15.89:0/963019376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 msgr2=0x557c9748d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 0x557c9748d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 -- 172.21.15.89:0/963019376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748c980 msgr2=0x557c9748cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748c980 0x557c9748cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 -- 172.21.15.89:0/963019376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c9725e5e0 con 0x557c9748dd00 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6019ed700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 0x557c9748d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb6029ef700 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009dc0 tx=0x7fb5f40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ff7b8d8213cab4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.677+0000 7fb5ecda2700 1 -- 172.21.15.89:0/963019376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4014610 con 0x557c9748dd00 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb5ecda2700 1 -- 172.21.15.89:0/963019376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5f400f050 con 0x557c9748dd00 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb5ecda2700 1 -- 172.21.15.89:0/963019376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4013960 con 0x557c9748dd00 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 -- 172.21.15.89:0/963019376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 msgr2=0x557c9748e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009dc0 tx=0x7fb5f40077e0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 -- 172.21.15.89:0/963019376 shutdown_connections 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748c980 0x557c9748cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748dd00 0x557c9748e180 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/963019376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748d2e0 0x557c9748d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 -- 172.21.15.89:0/963019376 >> 172.21.15.89:0/963019376 conn(0x557c97401090 msgr2=0x557c97487250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 -- 172.21.15.89:0/963019376 shutdown_connections 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.678+0000 7fb6105f6b80 1 -- 172.21.15.89:0/963019376 wait complete. 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 -- start start 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 Processor -- start 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97496360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748dd00 0x557c9749d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c973de940 con 0x557c9748d2e0 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9737cb60 con 0x557c9748dd00 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9737c860 con 0x557c9748c980 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6019ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6105f6b80 10 client.0 _ll_register_callbacks cb 0x557c972c6a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:17.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6019ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47788/0 (socket says 172.21.15.89:47788) 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6019ed700 1 -- 172.21.15.89:0/2848608355 learned_addr learned my addr 172.21.15.89:0/2848608355 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6029ef700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748dd00 0x557c9749d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6021ee700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97496360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.679+0000 7fb6019ed700 1 -- 172.21.15.89:0/2848608355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 msgr2=0x557c97496360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6019ed700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97496360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6019ed700 1 -- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748dd00 msgr2=0x557c9749d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6019ed700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748dd00 0x557c9749d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6019ed700 1 -- 172.21.15.89:0/2848608355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f4009a20 con 0x557c9748d2e0 2023-08-25T02:26:17.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6021ee700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97496360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:17.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6019ed700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0015000 tx=0x7fb5f0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9a664e52fc2b8b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0029070 con 0x557c9748d2e0 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb5f000f460 con 0x557c9748d2e0 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c9745c170 con 0x557c9748d2e0 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5f001d070 con 0x557c9748d2e0 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 10 client.15171 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:17.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557c9720ad90 con 0x557c9748d2e0 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557c9745c170 con 0x557c9748d2e0 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5f002e490 con 0x557c9748d2e0 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 10 client.15171 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 10 client.15171 resolve_mds: resolved name 'h' to daemon mds.h 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 20 client.15171 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:17.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c974a1ee0 0x557c974a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 4 client.15171 mds_command: new command op to 24470 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.680+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 --> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c97384b50 con 0x557c974a1ee0 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.681+0000 7fb6021ee700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c974a1ee0 0x557c974a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.681+0000 7fb5e5ffb700 10 client.15171 ms_handle_connect on v2:172.21.15.121:6836/2048790446 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.681+0000 7fb6021ee700 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c974a1ee0 0x557c974a0950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47ee3586b0409cc2 server_cookie=d4d5a4a259042947 in_seq=0 out_seq=0 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/2848608355 <== mds.2 v2:172.21.15.121:6836/2048790446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557c97384b50 con 0x557c974a1ee0 2023-08-25T02:26:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb5e5ffb700 10 client.15171 handle_command_reply: tid=0 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 client.15171 shutdown 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 msgr2=0x557c974968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0015000 tx=0x7fb5f0009310 comp rx=0 tx=0).stop 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 shutdown_connections 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c974a1ee0 0x557c974a0950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c9748dd00 0x557c9749d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97496360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb5e5ffb700 0 client.15171 ms_handle_reset on v2:172.21.15.121:6836/2048790446 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/2848608355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974968a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 >> 172.21.15.89:0/2848608355 conn(0x557c97401090 msgr2=0x557c97481af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 shutdown_connections 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 1 -- 172.21.15.89:0/2848608355 wait complete. 2023-08-25T02:26:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.682+0000 7fb6105f6b80 20 client.15171 trim_cache size 0 max 16384 2023-08-25T02:26:17.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.699+0000 7fb6105f6b80 1 Processor -- start 2023-08-25T02:26:17.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.699+0000 7fb6105f6b80 1 -- start start 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97490b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c9749d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c972608d0 con 0x557c9748d2e0 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c974948e0 con 0x557c97401090 2023-08-25T02:26:17.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9726cde0 con 0x557c9748c980 2023-08-25T02:26:17.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47798/0 (socket says 172.21.15.89:47798) 2023-08-25T02:26:17.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 -- 172.21.15.89:0/1712732970 learned_addr learned my addr 172.21.15.89:0/1712732970 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6019ed700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c9749d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6021ee700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97490b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 -- 172.21.15.89:0/1712732970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 msgr2=0x557c9749d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c9749d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 -- 172.21.15.89:0/1712732970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 msgr2=0x557c97490b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97490b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 -- 172.21.15.89:0/1712732970 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c971f3c10 con 0x557c9748d2e0 2023-08-25T02:26:17.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6019ed700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c9749d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:17.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb6029ef700 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007fd0 tx=0x7fb5f4009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4dbe330ad0c9e8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb5ed5a3700 1 -- 172.21.15.89:0/1712732970 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4022070 con 0x557c9748d2e0 2023-08-25T02:26:17.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb5ed5a3700 1 -- 172.21.15.89:0/1712732970 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5f4014410 con 0x557c9748d2e0 2023-08-25T02:26:17.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.700+0000 7fb5ed5a3700 1 -- 172.21.15.89:0/1712732970 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40052a0 con 0x557c9748d2e0 2023-08-25T02:26:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- 172.21.15.89:0/1712732970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 msgr2=0x557c974910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007fd0 tx=0x7fb5f4009990 comp rx=0 tx=0).stop 2023-08-25T02:26:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- 172.21.15.89:0/1712732970 shutdown_connections 2023-08-25T02:26:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97490b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974910c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/1712732970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c9749d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- 172.21.15.89:0/1712732970 >> 172.21.15.89:0/1712732970 conn(0x557c97495450 msgr2=0x557c97480ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- 172.21.15.89:0/1712732970 shutdown_connections 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- 172.21.15.89:0/1712732970 wait complete. 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- start start 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 Processor -- start 2023-08-25T02:26:17.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97498b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97499050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9749da90 con 0x557c9748d2e0 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9749dc00 con 0x557c97401090 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.701+0000 7fb6105f6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c974f5e10 con 0x557c9748c980 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6105f6b80 10 client.0 _ll_register_callbacks cb 0x7fb5f000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47804/0 (socket says 172.21.15.89:47804) 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6021ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97498b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 -- 172.21.15.89:0/160745514 learned_addr learned my addr 172.21.15.89:0/160745514 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6019ed700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97499050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 -- 172.21.15.89:0/160745514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 msgr2=0x557c97499050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97499050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 -- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 msgr2=0x557c97498b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97498b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 -- 172.21.15.89:0/160745514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c9725e5e0 con 0x557c9748d2e0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6029ef700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007820 tx=0x7fb5f4027650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b8a259b874bbfd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb5e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4022070 con 0x557c9748d2e0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c9747f860 con 0x557c9748d2e0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb5f4004ce0 con 0x557c9748d2e0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5f4004ea0 con 0x557c9748d2e0 2023-08-25T02:26:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6105f6b80 10 client.15177 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557c97231200 con 0x557c9748d2e0 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.702+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557c9747f860 con 0x557c9748d2e0 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5f402e030 con 0x557c9748d2e0 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 10 client.15177 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 10 client.15177 resolve_mds: resolved name 'h' to daemon mds.h 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 20 client.15177 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c9748dd00 0x557c974f7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 4 client.15177 mds_command: new command op to 24470 tid=0[{"prefix": "status"}] 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 --> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557c973d7920 con 0x557c9748dd00 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6021ee700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c9748dd00 0x557c974f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb5e5ffb700 10 client.15177 ms_handle_connect on v2:172.21.15.121:6836/2048790446 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6021ee700 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c9748dd00 0x557c974f7000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d2a2bad228bdf3a server_cookie=c0a503ed20b8bf9a in_seq=0 out_seq=0 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb5e5ffb700 1 -- 172.21.15.89:0/160745514 <== mds.2 v2:172.21.15.121:6836/2048790446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x557c973d7920 con 0x557c9748dd00 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb5e5ffb700 10 client.15177 handle_command_reply: tid=0 2023-08-25T02:26:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.703+0000 7fb6105f6b80 1 client.15177 shutdown 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 msgr2=0x557c974f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4007820 tx=0x7fb5f4027650 comp rx=0 tx=0).stop 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 shutdown_connections 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:6836/2048790446,v1:172.21.15.121:6837/2048790446] conn(0x557c9748dd00 0x557c974f7000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557c97401090 0x557c97498b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb5e5ffb700 0 client.15177 ms_handle_reset on v2:172.21.15.121:6836/2048790446 2023-08-25T02:26:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c9748d2e0 0x557c974f58d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 --2- 172.21.15.89:0/160745514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557c9748c980 0x557c97499050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 >> 172.21.15.89:0/160745514 conn(0x557c97495450 msgr2=0x557c97490b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 shutdown_connections 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 1 -- 172.21.15.89:0/160745514 wait complete. 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb6105f6b80 20 client.15177 trim_cache size 0 max 16384 2023-08-25T02:26:17.717 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24470, 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:26:17.718 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 69.073475248999998, 2023-08-25T02:26:17.719 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:26:17.719 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 82, 2023-08-25T02:26:17.719 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:26:17.719 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 87.157312848999993 2023-08-25T02:26:17.719 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:26:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 msgr2=0x7fb5e80a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 0x7fb5e80a2490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00095c0 tx=0x7fb5f0009530 comp rx=0 tx=0).stop 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 msgr2=0x7fb5fc196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800c920 tx=0x7fb5f800cce0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 shutdown_connections 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5fc106650 0x7fb5fc19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5fc1013f0 0x7fb5fc196610 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fb5e809ffd0 0x7fb5e80a2490 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 --2- 172.21.15.89:0/723729774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5fc103d20 0x7fb5fc196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 >> 172.21.15.89:0/723729774 conn(0x7fb5fc0fad80 msgr2=0x7fb5fc102060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 shutdown_connections 2023-08-25T02:26:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.704+0000 7fb604452700 1 -- 172.21.15.89:0/723729774 wait complete. 2023-08-25T02:26:17.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-08-25T02:26:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:17 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2772557944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:17.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:17 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1538510953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:17.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:17 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/147862638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:17.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3981967998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe78100a00 msgr2=0x7fbe78100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3981967998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe78100a00 0x7fbe78100e60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0099c0 tx=0x7fbe6c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3981967998 shutdown_connections 2023-08-25T02:26:17.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3981967998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe780ff810 0x7fbe780ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3981967998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe781013a0 0x7fbe78107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3981967998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe78100a00 0x7fbe78100e60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3981967998 >> 172.21.15.89:0/3981967998 conn(0x7fbe780fad80 msgr2=0x7fbe780fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3981967998 shutdown_connections 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3981967998 wait complete. 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.844+0000 7fbe7dbc8700 1 Processor -- start 2023-08-25T02:26:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- start start 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe780ff810 0x7fbe78195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe78100a00 0x7fbe78195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7819d080 con 0x7fbe781013a0 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7819d1c0 con 0x7fbe78100a00 2023-08-25T02:26:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7819d350 con 0x7fbe780ff810 2023-08-25T02:26:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47818/0 (socket says 172.21.15.89:47818) 2023-08-25T02:26:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 -- 172.21.15.89:0/3886653718 learned_addr learned my addr 172.21.15.89:0/3886653718 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe777fe700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe780ff810 0x7fbe78195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe78100a00 0x7fbe78195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 -- 172.21.15.89:0/3886653718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe780ff810 msgr2=0x7fbe78195500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe780ff810 0x7fbe78195500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 -- 172.21.15.89:0/3886653718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe78100a00 msgr2=0x7fbe78195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe78100a00 0x7fbe78195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 -- 172.21.15.89:0/3886653718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c0095f0 con 0x7fbe781013a0 2023-08-25T02:26:17.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe77fff700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbe70007ee0 tx=0x7fbe7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe74ff9700 1 -- 172.21.15.89:0/3886653718 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70017070 con 0x7fbe781013a0 2023-08-25T02:26:17.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe74ff9700 1 -- 172.21.15.89:0/3886653718 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe7000f040 con 0x7fbe781013a0 2023-08-25T02:26:17.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7819d630 con 0x7fbe781013a0 2023-08-25T02:26:17.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe74ff9700 1 -- 172.21.15.89:0/3886653718 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70005740 con 0x7fbe781013a0 2023-08-25T02:26:17.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.845+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7819db90 con 0x7fbe781013a0 2023-08-25T02:26:17.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.847+0000 7fbe74ff9700 1 -- 172.21.15.89:0/3886653718 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7fbe7000dc60 con 0x7fbe781013a0 2023-08-25T02:26:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.848+0000 7fbe74ff9700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 0x7fbe600a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.848+0000 7fbe74ff9700 1 -- 172.21.15.89:0/3886653718 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbe700ca9e0 con 0x7fbe781013a0 2023-08-25T02:26:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.848+0000 7fbe777fe700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 0x7fbe600a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.848+0000 7fbe777fe700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 0x7fbe600a2490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbe78100860 tx=0x7fbe6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 Processor -- start 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 -- start start 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f0cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f0e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477e72760 con 0x563477f0d4f0 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477e63730 con 0x563477f0cb90 2023-08-25T02:26:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477e61050 con 0x563477f0df10 2023-08-25T02:26:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47824/0 (socket says 172.21.15.89:47824) 2023-08-25T02:26:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe76ffd700 1 -- 172.21.15.89:0/1487205114 learned_addr learned my addr 172.21.15.89:0/1487205114 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.849+0000 7fbe77fff700 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f0e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe777fe700 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f0cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 -- 172.21.15.89:0/1487205114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 msgr2=0x563477f0e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f0e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 -- 172.21.15.89:0/1487205114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 msgr2=0x563477f0cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f0cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 -- 172.21.15.89:0/1487205114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563477cdc5e0 con 0x563477f0d4f0 2023-08-25T02:26:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0036f0 tx=0x7fbe6c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5279ba33217946f0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe5dffb700 1 -- 172.21.15.89:0/1487205114 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0052c0 con 0x563477f0d4f0 2023-08-25T02:26:17.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe5dffb700 1 -- 172.21.15.89:0/1487205114 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe6c01e040 con 0x563477f0d4f0 2023-08-25T02:26:17.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.850+0000 7fbe5dffb700 1 -- 172.21.15.89:0/1487205114 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c003bb0 con 0x563477f0d4f0 2023-08-25T02:26:17.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/1487205114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 msgr2=0x563477f0d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0036f0 tx=0x7fbe6c009f90 comp rx=0 tx=0).stop 2023-08-25T02:26:17.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/1487205114 shutdown_connections 2023-08-25T02:26:17.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f0cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f0e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/1487205114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f0d930 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/1487205114 >> 172.21.15.89:0/1487205114 conn(0x563477e7f090 msgr2=0x563477f05250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/1487205114 shutdown_connections 2023-08-25T02:26:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/1487205114 wait complete. 2023-08-25T02:26:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 -- start start 2023-08-25T02:26:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.851+0000 7fbe89d6cb80 1 Processor -- start 2023-08-25T02:26:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f16570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477e5c940 con 0x563477f0d4f0 2023-08-25T02:26:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477dfab60 con 0x563477f0cb90 2023-08-25T02:26:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477dfa860 con 0x563477f0df10 2023-08-25T02:26:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 10 client.0 _ll_register_callbacks cb 0x563477d44a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47830/0 (socket says 172.21.15.89:47830) 2023-08-25T02:26:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 -- 172.21.15.89:0/2243763647 learned_addr learned my addr 172.21.15.89:0/2243763647 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe777fe700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f16570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 -- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 msgr2=0x563477f1daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 -- 172.21.15.89:0/2243763647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 msgr2=0x563477f16570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f16570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 -- 172.21.15.89:0/2243763647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe781085a0 con 0x563477f0d4f0 2023-08-25T02:26:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0098e0 tx=0x7fbe6c00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b8fae81bddf76a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c003b30 con 0x563477f0d4f0 2023-08-25T02:26:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563477eda170 con 0x563477f0d4f0 2023-08-25T02:26:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbe6c005480 con 0x563477f0d4f0 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe6c003c90 con 0x563477f0d4f0 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.852+0000 7fbe89d6cb80 10 client.15189 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x563477c88d90 con 0x563477f0d4f0 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe6c03c080 con 0x563477f0d4f0 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe6c036030 con 0x563477f0d4f0 2023-08-25T02:26:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 10 client.15189 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 10 client.15189 resolve_mds: resolved name 'i' to daemon mds.i 2023-08-25T02:26:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 20 client.15189 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f21c80 0x563477f20a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 4 client.15189 mds_command: new command op to 24475 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:26:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 --> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563477e02b50 con 0x563477f21c80 2023-08-25T02:26:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe777fe700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f21c80 0x563477f20a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe56ffd700 10 client.15189 ms_handle_connect on v2:172.21.15.122:6838/3754052688 2023-08-25T02:26:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.853+0000 7fbe777fe700 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f21c80 0x563477f20a20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=626861f55c4e5d01 server_cookie=8e41b8d3c0960519 in_seq=0 out_seq=0 2023-08-25T02:26:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe56ffd700 1 -- 172.21.15.89:0/2243763647 <== mds.1 v2:172.21.15.122:6838/3754052688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563477e02b50 con 0x563477f21c80 2023-08-25T02:26:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe56ffd700 10 client.15189 handle_command_reply: tid=0 2023-08-25T02:26:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 client.15189 shutdown 2023-08-25T02:26:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 msgr2=0x563477f16ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0098e0 tx=0x7fbe6c00ba70 comp rx=0 tx=0).stop 2023-08-25T02:26:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 shutdown_connections 2023-08-25T02:26:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f21c80 0x563477f20a20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0cb90 0x563477f16570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe56ffd700 0 client.15189 ms_handle_reset on v2:172.21.15.122:6838/3754052688 2023-08-25T02:26:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0df10 0x563477f1daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/2243763647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f16ab0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 >> 172.21.15.89:0/2243763647 conn(0x563477e7f090 msgr2=0x563477effaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 shutdown_connections 2023-08-25T02:26:17.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.854+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/2243763647 wait complete. 2023-08-25T02:26:17.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.855+0000 7fbe89d6cb80 20 client.15189 trim_cache size 0 max 16384 2023-08-25T02:26:17.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 Processor -- start 2023-08-25T02:26:17.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 -- start start 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477e7f090 0x563477f0cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f1dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477cde8d0 con 0x563477f0d4f0 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477e1cdf0 con 0x563477e7f090 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477ceade0 con 0x563477f0cb90 2023-08-25T02:26:17.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47840/0 (socket says 172.21.15.89:47840) 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 -- 172.21.15.89:0/282824766 learned_addr learned my addr 172.21.15.89:0/282824766 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 -- 172.21.15.89:0/282824766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 msgr2=0x563477f1dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f1dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 -- 172.21.15.89:0/282824766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477e7f090 msgr2=0x563477f0cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477e7f090 0x563477f0cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 -- 172.21.15.89:0/282824766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563477c71c10 con 0x563477f0d4f0 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.872+0000 7fbe77fff700 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009510 tx=0x7fbe70083040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d387c31c9e196534 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe5e7fc700 1 -- 172.21.15.89:0/282824766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70098b10 con 0x563477f0d4f0 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe5e7fc700 1 -- 172.21.15.89:0/282824766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe70098410 con 0x563477f0d4f0 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe5e7fc700 1 -- 172.21.15.89:0/282824766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700058a0 con 0x563477f0d4f0 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/282824766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 msgr2=0x563477f20a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009510 tx=0x7fbe70083040 comp rx=0 tx=0).stop 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/282824766 shutdown_connections 2023-08-25T02:26:17.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477e7f090 0x563477f0cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477f0d4f0 0x563477f20a20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/282824766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f1dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/282824766 >> 172.21.15.89:0/282824766 conn(0x563477f05580 msgr2=0x563477f15270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/282824766 shutdown_connections 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/282824766 wait complete. 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.873+0000 7fbe89d6cb80 1 -- start start 2023-08-25T02:26:17.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 Processor -- start 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f193c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 0x563477f1c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477f1c920 con 0x563477e7f090 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477f1ca90 con 0x563477f0d4f0 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563477f1cc50 con 0x563477f0cb90 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe89d6cb80 10 client.0 _ll_register_callbacks cb 0x7fbe6c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:17.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47842/0 (socket says 172.21.15.89:47842) 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 -- 172.21.15.89:0/4138102709 learned_addr learned my addr 172.21.15.89:0/4138102709 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 -- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 msgr2=0x563477f193c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f193c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 -- 172.21.15.89:0/4138102709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 msgr2=0x563477f1c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:17.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe77fff700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 0x563477f1c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 0x563477f1c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 -- 172.21.15.89:0/4138102709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7819db90 con 0x563477e7f090 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe77fff700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 0x563477f1c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe777fe700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800be80 tx=0x7fbe68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9369f4abd50dbfa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.874+0000 7fbe56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68024070 con 0x563477e7f090 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbe68017070 con 0x563477e7f090 2023-08-25T02:26:17.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563477efd860 con 0x563477e7f090 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe68011d70 con 0x563477e7f090 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 10 client.15195 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x563477caf200 con 0x563477e7f090 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe68037080 con 0x563477e7f090 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe68031030 con 0x563477e7f090 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 10 client.15195 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:17.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 10 client.15195 resolve_mds: resolved name 'i' to daemon mds.i 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 20 client.15195 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f0df10 0x563477f77a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 4 client.15195 mds_command: new command op to 24475 tid=0[{"prefix": "status"}] 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 --> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563477e55920 con 0x563477f0df10 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.875+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f0df10 0x563477f77a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe56ffd700 10 client.15195 ms_handle_connect on v2:172.21.15.122:6838/3754052688 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe76ffd700 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f0df10 0x563477f77a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4052ff8d10f60e3 server_cookie=6b148eed79168f3c in_seq=0 out_seq=0 2023-08-25T02:26:17.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe56ffd700 1 -- 172.21.15.89:0/4138102709 <== mds.1 v2:172.21.15.122:6838/3754052688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x563477e55920 con 0x563477f0df10 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe56ffd700 10 client.15195 handle_command_reply: tid=0 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 client.15195 shutdown 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 msgr2=0x563477f18e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800be80 tx=0x7fbe68004750 comp rx=0 tx=0).stop 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 shutdown_connections 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] conn(0x563477f0df10 0x563477f77a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563477f0d4f0 0x563477f1c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe56ffd700 0 client.15195 ms_handle_reset on v2:172.21.15.122:6838/3754052688 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563477e7f090 0x563477f18e80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 --2- 172.21.15.89:0/4138102709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563477f0cb90 0x563477f193c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 >> 172.21.15.89:0/4138102709 conn(0x563477f05580 msgr2=0x563477f20a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 shutdown_connections 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 1 -- 172.21.15.89:0/4138102709 wait complete. 2023-08-25T02:26:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.876+0000 7fbe89d6cb80 20 client.15195 trim_cache size 0 max 16384 2023-08-25T02:26:17.902 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:26:17.902 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:26:17.902 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2023-08-25T02:26:17.902 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24475, 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 70.254127656999998, 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 82, 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:26:17.903 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 85.169565173999999 2023-08-25T02:26:17.904 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:26:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 msgr2=0x7fbe600a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 0x7fbe600a2490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbe78100860 tx=0x7fbe6800b410 comp rx=0 tx=0).stop 2023-08-25T02:26:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 msgr2=0x7fbe7819cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:17.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbe70007ee0 tx=0x7fbe7000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 shutdown_connections 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe781013a0 0x7fbe7819cab0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe780ff810 0x7fbe78195500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbe6009ffd0 0x7fbe600a2490 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 --2- 172.21.15.89:0/3886653718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe78100a00 0x7fbe78195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 >> 172.21.15.89:0/3886653718 conn(0x7fbe780fad80 msgr2=0x7fbe780fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 shutdown_connections 2023-08-25T02:26:17.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:17.877+0000 7fbe7dbc8700 1 -- 172.21.15.89:0/3886653718 wait complete. 2023-08-25T02:26:17.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-08-25T02:26:17.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:17 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2772557944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:17.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:17 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1538510953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:17.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:17 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/147862638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:17.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:17 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2772557944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:26:17.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:17 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1538510953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:26:17.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:17 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/147862638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:26:18.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 -- 172.21.15.89:0/1008177529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 msgr2=0x7f969c100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/1008177529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c100e60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f968c0099c0 tx=0x7f968c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 -- 172.21.15.89:0/1008177529 shutdown_connections 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/1008177529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c0ff810 0x7f969c0ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/1008177529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c1013a0 0x7f969c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/1008177529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c100e60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 -- 172.21.15.89:0/1008177529 >> 172.21.15.89:0/1008177529 conn(0x7f969c0fad80 msgr2=0x7f969c0fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 -- 172.21.15.89:0/1008177529 shutdown_connections 2023-08-25T02:26:18.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.029+0000 7f96a0b5f700 1 -- 172.21.15.89:0/1008177529 wait complete. 2023-08-25T02:26:18.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 Processor -- start 2023-08-25T02:26:18.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- start start 2023-08-25T02:26:18.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 0x7f969c1037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 0x7f969c102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c102ac0 con 0x7f969c100a00 2023-08-25T02:26:18.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c102c30 con 0x7f969c1013a0 2023-08-25T02:26:18.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c1a06a0 con 0x7f969c0ff810 2023-08-25T02:26:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47858/0 (socket says 172.21.15.89:47858) 2023-08-25T02:26:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 -- 172.21.15.89:0/2472724311 learned_addr learned my addr 172.21.15.89:0/2472724311 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f969ad9d700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 0x7f969c102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f969a59c700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 0x7f969c1037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 -- 172.21.15.89:0/2472724311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 msgr2=0x7f969c1037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 0x7f969c1037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 -- 172.21.15.89:0/2472724311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 msgr2=0x7f969c102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 0x7f969c102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 -- 172.21.15.89:0/2472724311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0095f0 con 0x7f969c100a00 2023-08-25T02:26:18.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f969a59c700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 0x7f969c1037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:18.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f969ad9d700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 0x7f969c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:18.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f9699d9b700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f968c009990 tx=0x7f968c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f968b7fe700 1 -- 172.21.15.89:0/2472724311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c01d070 con 0x7f969c100a00 2023-08-25T02:26:18.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c1a0920 con 0x7f969c100a00 2023-08-25T02:26:18.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f968b7fe700 1 -- 172.21.15.89:0/2472724311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f968c022470 con 0x7f969c100a00 2023-08-25T02:26:18.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f968b7fe700 1 -- 172.21.15.89:0/2472724311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c004230 con 0x7f969c100a00 2023-08-25T02:26:18.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.030+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c1a0dd0 con 0x7f969c100a00 2023-08-25T02:26:18.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.032+0000 7f968b7fe700 1 -- 172.21.15.89:0/2472724311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f968c0225e0 con 0x7f969c100a00 2023-08-25T02:26:18.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.033+0000 7f968b7fe700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 0x7f96840a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.033+0000 7f969a59c700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 0x7f96840a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.033+0000 7f968b7fe700 1 -- 172.21.15.89:0/2472724311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f968c0cc520 con 0x7f969c100a00 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.033+0000 7f969a59c700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 0x7f96840a2490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9690005fd0 tx=0x7f9690005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 Processor -- start 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 -- start start 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e67b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e5e2760 con 0x55c90e67a980 2023-08-25T02:26:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e5d3730 con 0x55c90e67b2e0 2023-08-25T02:26:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f96acd03b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e5d1050 con 0x55c90e67bd00 2023-08-25T02:26:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47864/0 (socket says 172.21.15.89:47864) 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f969a59c700 1 -- 172.21.15.89:0/2722360859 learned_addr learned my addr 172.21.15.89:0/2722360859 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.034+0000 7f9699d9b700 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e67b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 -- 172.21.15.89:0/2722360859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 msgr2=0x55c90e67c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 -- 172.21.15.89:0/2722360859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 msgr2=0x55c90e67b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e67b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 -- 172.21.15.89:0/2722360859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c90e44c5e0 con 0x55c90e67a980 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f9699d9b700 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e67b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f969a59c700 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f969000edb0 tx=0x7f9690011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c566448c64a6cf70 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f9688ff9700 1 -- 172.21.15.89:0/2722360859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690011c90 con 0x55c90e67a980 2023-08-25T02:26:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f9688ff9700 1 -- 172.21.15.89:0/2722360859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9690011df0 con 0x55c90e67a980 2023-08-25T02:26:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f9688ff9700 1 -- 172.21.15.89:0/2722360859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690010710 con 0x55c90e67a980 2023-08-25T02:26:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f96acd03b80 1 -- 172.21.15.89:0/2722360859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 msgr2=0x55c90e67ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.035+0000 7f96acd03b80 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f969000edb0 tx=0x7f9690011000 comp rx=0 tx=0).stop 2023-08-25T02:26:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- 172.21.15.89:0/2722360859 shutdown_connections 2023-08-25T02:26:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e67ada0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- 172.21.15.89:0/2722360859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e67b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- 172.21.15.89:0/2722360859 >> 172.21.15.89:0/2722360859 conn(0x55c90e5ef090 msgr2=0x55c90e675250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- 172.21.15.89:0/2722360859 shutdown_connections 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- 172.21.15.89:0/2722360859 wait complete. 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- start start 2023-08-25T02:26:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 Processor -- start 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67a980 0x55c90e6843b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e68b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e5cc940 con 0x55c90e67b2e0 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e56ab60 con 0x55c90e67a980 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e56a860 con 0x55c90e67bd00 2023-08-25T02:26:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f96acd03b80 10 client.0 _ll_register_callbacks cb 0x55c90e4b4a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f9699d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47872/0 (socket says 172.21.15.89:47872) 2023-08-25T02:26:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.036+0000 7f9699d9b700 1 -- 172.21.15.89:0/1469273704 learned_addr learned my addr 172.21.15.89:0/1469273704 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f969ad9d700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e68b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f969a59c700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67a980 0x55c90e6843b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 -- 172.21.15.89:0/1469273704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 msgr2=0x55c90e68b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e68b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 -- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67a980 msgr2=0x55c90e6843b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67a980 0x55c90e6843b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 -- 172.21.15.89:0/1469273704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969000ea90 con 0x55c90e67b2e0 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f969ad9d700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e68b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9699d9b700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f968c099c80 tx=0x7f968c022bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327d871ed46ea30d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0cc2d0 con 0x55c90e67b2e0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f968c0a1d00 con 0x55c90e67b2e0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c90e64a170 con 0x55c90e67b2e0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f968c099470 con 0x55c90e67b2e0 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 10 client.15207 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c90e3f8d90 con 0x55c90e67b2e0 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c90e64a170 con 0x55c90e67b2e0 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f968c094b90 con 0x55c90e67b2e0 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 10 client.15207 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 10 client.15207 resolve_mds: resolved name 'b' to daemon mds.b 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 20 client.15207 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e68e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 4 client.15207 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:26:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.037+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c90e572b50 con 0x55c90e68fa20 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.038+0000 7f969a59c700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e68e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.038+0000 7f9681ffb700 10 client.15207 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.038+0000 7f969a59c700 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e68e7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5de7931ca561423 server_cookie=617ddb0d957ae6a6 in_seq=0 out_seq=0 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f9681ffb700 1 -- 172.21.15.89:0/1469273704 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c90e572b50 con 0x55c90e68fa20 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f9681ffb700 10 client.15207 handle_command_reply: tid=0 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 client.15207 shutdown 2023-08-25T02:26:18.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 msgr2=0x55c90e6848f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f968c099c80 tx=0x7f968c022bc0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 shutdown_connections 2023-08-25T02:26:18.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e68e7d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67a980 0x55c90e6843b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f9681ffb700 0 client.15207 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e68b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 --2- 172.21.15.89:0/1469273704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67b2e0 0x55c90e6848f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 >> 172.21.15.89:0/1469273704 conn(0x55c90e5ef090 msgr2=0x55c90e66faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 shutdown_connections 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 1 -- 172.21.15.89:0/1469273704 wait complete. 2023-08-25T02:26:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.039+0000 7f96acd03b80 20 client.15207 trim_cache size 0 max 16384 2023-08-25T02:26:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 Processor -- start 2023-08-25T02:26:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 -- start start 2023-08-25T02:26:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e68b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e44e8d0 con 0x55c90e67a980 2023-08-25T02:26:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e6855a0 con 0x55c90e67b2e0 2023-08-25T02:26:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96acd03b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e671830 con 0x55c90e67bd00 2023-08-25T02:26:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47878/0 (socket says 172.21.15.89:47878) 2023-08-25T02:26:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 -- 172.21.15.89:0/364063006 learned_addr learned my addr 172.21.15.89:0/364063006 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969ad9d700 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 -- 172.21.15.89:0/364063006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 msgr2=0x55c90e67ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 -- 172.21.15.89:0/364063006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 msgr2=0x55c90e68b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e68b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 -- 172.21.15.89:0/364063006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c90e698650 con 0x55c90e67a980 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f969a59c700 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9690013040 tx=0x7f9690014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e98cd3e8932f240 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96897fa700 1 -- 172.21.15.89:0/364063006 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969002e040 con 0x55c90e67a980 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96897fa700 1 -- 172.21.15.89:0/364063006 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9690011e10 con 0x55c90e67a980 2023-08-25T02:26:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.057+0000 7f96897fa700 1 -- 172.21.15.89:0/364063006 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96900052a0 con 0x55c90e67a980 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- 172.21.15.89:0/364063006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 msgr2=0x55c90e68e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9690013040 tx=0x7f9690014000 comp rx=0 tx=0).stop 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- 172.21.15.89:0/364063006 shutdown_connections 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e67ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e68e7d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- 172.21.15.89:0/364063006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e68b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- 172.21.15.89:0/364063006 >> 172.21.15.89:0/364063006 conn(0x55c90e5ef090 msgr2=0x55c90e683060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- 172.21.15.89:0/364063006 shutdown_connections 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- 172.21.15.89:0/364063006 wait complete. 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- start start 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 Processor -- start 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e6870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e6e3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e68b9d0 con 0x55c90e67a980 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e68bb40 con 0x55c90e67b2e0 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.058+0000 7f96acd03b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90e6e3950 con 0x55c90e67bd00 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f96acd03b80 10 client.0 _ll_register_callbacks cb 0x7f968c099ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47884/0 (socket says 172.21.15.89:47884) 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 -- 172.21.15.89:0/4048125894 learned_addr learned my addr 172.21.15.89:0/4048125894 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969ad9d700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e6e3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9699d9b700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e6870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 -- 172.21.15.89:0/4048125894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 msgr2=0x55c90e6e3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e6e3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 -- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 msgr2=0x55c90e6870a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e6870a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 -- 172.21.15.89:0/4048125894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c90e44c5e0 con 0x55c90e67a980 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9699d9b700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e6870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f969a59c700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f969000e990 tx=0x7f9690011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f1fdb7a2c8f726 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9681ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690031c80 con 0x55c90e67a980 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9690031de0 con 0x55c90e67a980 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c90e66d860 con 0x55c90e67a980 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9690005030 con 0x55c90e67a980 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f96acd03b80 10 client.15213 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c90e3e1c10 con 0x55c90e67a980 2023-08-25T02:26:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.059+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c90e66d860 con 0x55c90e67a980 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9690040020 con 0x55c90e67a980 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 10 client.15213 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 10 client.15213 resolve_mds: resolved name 'b' to daemon mds.b 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 20 client.15213 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e6e4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 4 client.15213 mds_command: new command op to 24509 tid=0[{"prefix": "status"}] 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c90e5c5920 con 0x55c90e68fa20 2023-08-25T02:26:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9699d9b700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e6e4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9681ffb700 10 client.15213 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9699d9b700 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e6e4a90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=175cca78bc6ecb98 server_cookie=5b826870af256821 in_seq=0 out_seq=0 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9681ffb700 1 -- 172.21.15.89:0/4048125894 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c90e5c5920 con 0x55c90e68fa20 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.060+0000 7f9681ffb700 10 client.15213 handle_command_reply: tid=0 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 client.15213 shutdown 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 msgr2=0x55c90e686b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f969000e990 tx=0x7f9690011b90 comp rx=0 tx=0).stop 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 shutdown_connections 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c90e68fa20 0x55c90e6e4a90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c90e67bd00 0x55c90e6e3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f9681ffb700 0 client.15213 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c90e67a980 0x55c90e686b60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 --2- 172.21.15.89:0/4048125894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c90e67b2e0 0x55c90e6870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 >> 172.21.15.89:0/4048125894 conn(0x55c90e5ef090 msgr2=0x55c90e68e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 shutdown_connections 2023-08-25T02:26:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 1 -- 172.21.15.89:0/4048125894 wait complete. 2023-08-25T02:26:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96acd03b80 20 client.15213 trim_cache size 0 max 16384 2023-08-25T02:26:18.075 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:26:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "7eb69934-42ed-11ee-9b3d-001a4aab830c", 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24509, 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 73.460528198999995, 2023-08-25T02:26:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 22, 2023-08-25T02:26:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 82, 2023-08-25T02:26:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 80, 2023-08-25T02:26:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 78.680022574000006 2023-08-25T02:26:18.077 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:26:18.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 msgr2=0x7f96840a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 0x7f96840a2490 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9690005fd0 tx=0x7f9690005f00 comp rx=0 tx=0).stop 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 msgr2=0x7f969c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f968c009990 tx=0x7f968c005e80 comp rx=0 tx=0).stop 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 shutdown_connections 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f968409ffd0 0x7f96840a2490 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c1013a0 0x7f969c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f969c0ff810 0x7f969c1037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 --2- 172.21.15.89:0/2472724311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f969c100a00 0x7f969c101de0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:18.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 >> 172.21.15.89:0/2472724311 conn(0x7f969c0fad80 msgr2=0x7f969c067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:18.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 shutdown_connections 2023-08-25T02:26:18.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:18.061+0000 7f96a0b5f700 1 -- 172.21.15.89:0/2472724311 wait complete. 2023-08-25T02:26:18.080 INFO:teuthology.run_tasks:Running task kclient... 2023-08-25T02:26:18.109 INFO:tasks.kclient:Mounting kernel clients... 2023-08-25T02:26:18.109 INFO:tasks.kclient:config is {'client.0': {}} 2023-08-25T02:26:18.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:18.110 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-08-25T02:26:18.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:18.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-08-25T02:26:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-08-25T02:26:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:26:18.200 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mountpoint': '/tmp/cephfs'} 2023-08-25T02:26:18.201 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-08-25T02:26:18.202 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2023-08-25T02:26:18.202 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2023-08-25T02:26:18.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph fs ls 2023-08-25T02:26:18.791 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:26:18.962 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:18 smithi089 ceph-mon[98732]: pgmap v289: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 100 B/s, 0 objects/s recovering 2023-08-25T02:26:18.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:18 smithi121 ceph-mon[100099]: pgmap v289: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 100 B/s, 0 objects/s recovering 2023-08-25T02:26:18.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:18 smithi122 ceph-mon[100457]: pgmap v289: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 100 B/s, 0 objects/s recovering 2023-08-25T02:26:19.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 -- 172.21.15.89:0/3318896029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e000ffbc0 msgr2=0x7f0e00103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:19.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 --2- 172.21.15.89:0/3318896029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e000ffbc0 0x7f0e00103c10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009990 tx=0x7f0df0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 -- 172.21.15.89:0/3318896029 shutdown_connections 2023-08-25T02:26:19.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 --2- 172.21.15.89:0/3318896029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 0x7f0e000ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 --2- 172.21.15.89:0/3318896029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e001042f0 0x7f0e0010bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 --2- 172.21.15.89:0/3318896029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e000ffbc0 0x7f0e00103c10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 -- 172.21.15.89:0/3318896029 >> 172.21.15.89:0/3318896029 conn(0x7f0e00075d90 msgr2=0x7f0e000fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 -- 172.21.15.89:0/3318896029 shutdown_connections 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.969+0000 7f0e07938700 1 -- 172.21.15.89:0/3318896029 wait complete. 2023-08-25T02:26:19.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 Processor -- start 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 -- start start 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 0x7f0e0019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e000ffbc0 0x7f0e0019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00196ee0 con 0x7f0e001042f0 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00197050 con 0x7f0e000ff210 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e07938700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00197210 con 0x7f0e000ffbc0 2023-08-25T02:26:19.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:19.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47902/0 (socket says 172.21.15.89:47902) 2023-08-25T02:26:19.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 -- 172.21.15.89:0/2126613356 learned_addr learned my addr 172.21.15.89:0/2126613356 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:19.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e04ed3700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e000ffbc0 0x7f0e0019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:19.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e056d4700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 0x7f0e0019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:19.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 -- 172.21.15.89:0/2126613356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e000ffbc0 msgr2=0x7f0e0019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:19.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e000ffbc0 0x7f0e0019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 -- 172.21.15.89:0/2126613356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 msgr2=0x7f0e0019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:19.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 0x7f0e0019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:19.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.970+0000 7f0e05ed5700 1 -- 172.21.15.89:0/2126613356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00095f0 con 0x7f0e001042f0 2023-08-25T02:26:19.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0e05ed5700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007ee0 tx=0x7f0df800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:19.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8017070 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0df800f040 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e001974f0 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8005580 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.971+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e001a37e0 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.972+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0004e620 con 0x7f0e001042f0 2023-08-25T02:26:19.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.973+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f0df800db20 con 0x7f0e001042f0 2023-08-25T02:26:19.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.973+0000 7f0df67fc700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 0x7f0dec0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:19.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.973+0000 7f0e056d4700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 0x7f0dec0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:19.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.973+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f0df80ca8b0 con 0x7f0e001042f0 2023-08-25T02:26:19.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.974+0000 7f0e056d4700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 0x7f0dec0a2490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005950 tx=0x7f0dfc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:19.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:19.978+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0df8094620 con 0x7f0e001042f0 2023-08-25T02:26:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.208+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0e0010f940 con 0x7f0e001042f0 2023-08-25T02:26:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.209+0000 7f0df67fc700 1 -- 172.21.15.89:0/2126613356 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0df8097c40 con 0x7f0e001042f0 2023-08-25T02:26:20.210 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-08-25T02:26:20.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 msgr2=0x7f0dec0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:20.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 0x7f0dec0a2490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005950 tx=0x7f0dfc00b410 comp rx=0 tx=0).stop 2023-08-25T02:26:20.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 msgr2=0x7f0e00196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007ee0 tx=0x7f0df800d3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 shutdown_connections 2023-08-25T02:26:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f0dec09ffd0 0x7f0dec0a2490 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e000ff210 0x7f0e0019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e001042f0 0x7f0e00196870 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 --2- 172.21.15.89:0/2126613356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e000ffbc0 0x7f0e0019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 >> 172.21.15.89:0/2126613356 conn(0x7f0e00075d90 msgr2=0x7f0e00101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 shutdown_connections 2023-08-25T02:26:20.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:20.211+0000 7f0e07938700 1 -- 172.21.15.89:0/2126613356 wait complete. 2023-08-25T02:26:20.544 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-08-25T02:26:20.544 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-08-25T02:26:20.545 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-08-25T02:26:20.545 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-08-25T02:26:20.546 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2023-08-25T02:26:20.546 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-08-25T02:26:20.547 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-08-25T02:26:20.547 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-08-25T02:26:20.548 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2023-08-25T02:26:20.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:20.548 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-08-25T02:26:20.567 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-08-25T02:26:20.568 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-08-25T02:26:20.568 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-08-25T02:26:20.568 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-08-25T02:26:20.568 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-08-25T02:26:20.568 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-08-25T02:26:20.569 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-08-25T02:26:20.570 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-08-25T02:26:20.570 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41659sec preferred_lft 41659sec 2023-08-25T02:26:20.570 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::24ab:d266:e6a3:caa8/64 scope link noprefixroute 2023-08-25T02:26:20.570 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-08-25T02:26:20.570 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-08-25T02:26:20.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:20.571 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-08-25T02:26:20.571 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-08-25T02:26:20.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2023-08-25T02:26:20.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2023-08-25T02:26:20.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2023-08-25T02:26:20.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-08-25T02:26:20.573 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-08-25T02:26:20.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:20.742 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-08-25T02:26:20.770 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-08-25T02:26:20.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:20.772 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-08-25T02:26:20.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:20 smithi089 ceph-mon[98732]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 51 B/s, 0 objects/s recovering 2023-08-25T02:26:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:20 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2126613356' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-08-25T02:26:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-08-25T02:26:20.873 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-08-25T02:26:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-08-25T02:26:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-08-25T02:26:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-08-25T02:26:20.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:20.875 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-08-25T02:26:20.879 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-08-25T02:26:20.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-08-25T02:26:20.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-08-25T02:26:20.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-08-25T02:26:20.880 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-08-25T02:26:20.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:20 smithi121 ceph-mon[100099]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 51 B/s, 0 objects/s recovering 2023-08-25T02:26:20.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:20 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2126613356' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-08-25T02:26:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:20 smithi122 ceph-mon[100457]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 51 B/s, 0 objects/s recovering 2023-08-25T02:26:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:20 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2126613356' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-08-25T02:26:21.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.123 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-08-25T02:26:21.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.142 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2023-08-25T02:26:21.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.202 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-08-25T02:26:21.202 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-08-25T02:26:21.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--tmp-cephfs 2023-08-25T02:26:21.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--tmp-cephfs 0 2023-08-25T02:26:21.203 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-08-25T02:26:21.296 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2023-08-25T02:26:21.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.297 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-08-25T02:26:21.297 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-08-25T02:26:21.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2023-08-25T02:26:21.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-08-25T02:26:21.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2023-08-25T02:26:21.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2023-08-25T02:26:21.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2023-08-25T02:26:21.298 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-08-25T02:26:21.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.516 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-08-25T02:26:21.516 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-08-25T02:26:21.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-08-25T02:26:21.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2023-08-25T02:26:21.517 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-08-25T02:26:21.615 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T02:26:21.615 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /tmp/cephfs 2023-08-25T02:26:21.634 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/tmp/cephfs' 2023-08-25T02:26:21.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T02:26:21.635 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /tmp/cephfs 2023-08-25T02:26:21.691 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-08-25T02:26:21.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.692 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context=system_u:object_r:postgresql_db_t:s0 2023-08-25T02:26:21.786 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2023-08-25T02:26:21.787 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2023-08-25T02:26:21.787 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-08-25T02:26:21.922 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2023-08-25T02:26:21.923 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2023-08-25T02:26:21.923 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-08-25T02:26:21.924 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.121:3300,172.21.15.122:3300" 2023-08-25T02:26:21.924 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:3300,172.21.15.121:3300,172.21.15.122:3300:/ 2023-08-25T02:26:21.924 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=7eb69934-42ed-11ee-9b3d-001a4aab830c" will pass to kernel 2023-08-25T02:26:21.925 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-08-25T02:26:21.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /tmp/cephfs 2023-08-25T02:26:21.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:21.954 DEBUG:teuthology.orchestra.run.smithi089:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-08-25T02:26:22.043 INFO:teuthology.orchestra.run.smithi089.stdout:client15222 2023-08-25T02:26:22.044 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:26:22.045 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T02:26:22.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:22.066 DEBUG:teuthology.orchestra.run.smithi089:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-08-25T02:26:22.153 INFO:teuthology.orchestra.run.smithi089.stdout:client15222 2023-08-25T02:26:22.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:22.155 DEBUG:teuthology.orchestra.run.smithi089:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-08-25T02:26:22.202 INFO:teuthology.orchestra.run.smithi089.stdout:client15222 2023-08-25T02:26:22.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:26:22.203 DEBUG:teuthology.orchestra.run.smithi089:> (cd /tmp/cephfs && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c.client15222/status') 2023-08-25T02:26:22.258 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.15222 (3)192.168.0.1:0/2337416110 2023-08-25T02:26:22.258 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2023-08-25T02:26:22.259 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2023-08-25T02:26:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2023-08-25T02:26:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:70 bytes copied, 2.1938e-05 s, 3.2 MB/s 2023-08-25T02:26:22.261 INFO:teuthology.run_tasks:Running task check-counter... 2023-08-25T02:26:22.339 INFO:teuthology.run_tasks:Running task exec... 2023-08-25T02:26:22.358 INFO:teuthology.task.exec:Executing custom commands... 2023-08-25T02:26:22.359 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:26:22.359 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-08-25T02:26:22.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1475129230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f640ff840 msgr2=0x7f5f640ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1475129230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f640ff840 0x7f5f640ffc60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5f540099c0 tx=0x7f5f54009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1475129230 shutdown_connections 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1475129230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f641013d0 0x7f5f64107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1475129230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f640ff840 0x7f5f640ffc60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1475129230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f64100a30 0x7f5f64100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.480+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1475129230 >> 172.21.15.89:0/1475129230 conn(0x7f5f640fadb0 msgr2=0x7f5f640fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1475129230 shutdown_connections 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1475129230 wait complete. 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 Processor -- start 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- start start 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f640ff840 0x7f5f641989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f64100a30 0x7f5f64198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:22.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6419a3d0 con 0x7f5f641013d0 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6419a540 con 0x7f5f64100a30 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f6b6b0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6419a700 con 0x7f5f640ff840 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f69c4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f69c4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47920/0 (socket says 172.21.15.89:47920) 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f69c4d700 1 -- 172.21.15.89:0/1293323385 learned_addr learned my addr 172.21.15.89:0/1293323385 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.481+0000 7f5f68c4b700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f64100a30 0x7f5f64198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:22.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f6944c700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f640ff840 0x7f5f641989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 -- 172.21.15.89:0/1293323385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f640ff840 msgr2=0x7f5f641989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f640ff840 0x7f5f641989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 -- 172.21.15.89:0/1293323385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f64100a30 msgr2=0x7f5f64198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f64100a30 0x7f5f64198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 -- 172.21.15.89:0/1293323385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540095f0 con 0x7f5f641013d0 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f69c4d700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00ada0 tx=0x7f5f5c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:22.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c00f070 con 0x7f5f641013d0 2023-08-25T02:26:22.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f5c007b30 con 0x7f5f641013d0 2023-08-25T02:26:22.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c005280 con 0x7f5f641013d0 2023-08-25T02:26:22.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6419a9e0 con 0x7f5f641013d0 2023-08-25T02:26:22.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.482+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641a11a0 con 0x7f5f641013d0 2023-08-25T02:26:22.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.483+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6404e700 con 0x7f5f641013d0 2023-08-25T02:26:22.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.484+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 134671+0+0 (secure 0 0 0) 0x7f5f5c00c8d0 con 0x7f5f641013d0 2023-08-25T02:26:22.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.485+0000 7f5f5a7fc700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 0x7f5f500a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:22.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.485+0000 7f5f6944c700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 0x7f5f500a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:22.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.485+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5f5c0ca7c0 con 0x7f5f641013d0 2023-08-25T02:26:22.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.485+0000 7f5f6944c700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 0x7f5f500a2510 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009f50 tx=0x7f5f5400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:22.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.489+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5f5c097ff0 con 0x7f5f641013d0 2023-08-25T02:26:22.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:22.739+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5f641a14b0 con 0x7f5f641013d0 2023-08-25T02:26:22.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:22 smithi121 ceph-mon[100099]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-08-25T02:26:22.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:22 smithi122 ceph-mon[100457]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-08-25T02:26:23.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:22 smithi089 ceph-mon[98732]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-08-25T02:26:23.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.665+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 17) v1 ==== 134688+0+0 (secure 0 0 0) 0x7f5f5c00a090 con 0x7f5f641013d0 2023-08-25T02:26:23.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.666+0000 7f5f5a7fc700 1 -- 172.21.15.89:0/1293323385 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5f5c096360 con 0x7f5f641013d0 2023-08-25T02:26:23.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 msgr2=0x7f5f500a2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:23.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 0x7f5f500a2510 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009f50 tx=0x7f5f5400b540 comp rx=0 tx=0).stop 2023-08-25T02:26:23.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 msgr2=0x7f5f64199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:23.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00ada0 tx=0x7f5f5c00c330 comp rx=0 tx=0).stop 2023-08-25T02:26:23.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 shutdown_connections 2023-08-25T02:26:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f641013d0 0x7f5f64199d70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f640ff840 0x7f5f641989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f5f500a0050 0x7f5f500a2510 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 --2- 172.21.15.89:0/1293323385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f64100a30 0x7f5f64198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 >> 172.21.15.89:0/1293323385 conn(0x7f5f640fadb0 msgr2=0x7f5f64068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 shutdown_connections 2023-08-25T02:26:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.669+0000 7f5f6b6b0700 1 -- 172.21.15.89:0/1293323385 wait complete. 2023-08-25T02:26:23.689 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-08-25T02:26:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 -- 172.21.15.89:0/1915636910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c0ff1c0 msgr2=0x7f656c0ff5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 --2- 172.21.15.89:0/1915636910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c0ff1c0 0x7f656c0ff5e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6568009990 tx=0x7f6568009ca0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 -- 172.21.15.89:0/1915636910 shutdown_connections 2023-08-25T02:26:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 --2- 172.21.15.89:0/1915636910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c1004e0 0x7f656c100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 --2- 172.21.15.89:0/1915636910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c0ff1c0 0x7f656c0ff5e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 --2- 172.21.15.89:0/1915636910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 0x7f656c0fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 -- 172.21.15.89:0/1915636910 >> 172.21.15.89:0/1915636910 conn(0x7f656c0fada0 msgr2=0x7f656c0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:23.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 -- 172.21.15.89:0/1915636910 shutdown_connections 2023-08-25T02:26:23.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.819+0000 7f65742c0700 1 -- 172.21.15.89:0/1915636910 wait complete. 2023-08-25T02:26:23.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 Processor -- start 2023-08-25T02:26:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 -- start start 2023-08-25T02:26:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c0ff1c0 0x7f656c1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 0x7f656c198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19a2f0 con 0x7f656c1004e0 2023-08-25T02:26:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19a460 con 0x7f656c0ff1c0 2023-08-25T02:26:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f65742c0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19a620 con 0x7f656c0ffb20 2023-08-25T02:26:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:23.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47974/0 (socket says 172.21.15.89:47974) 2023-08-25T02:26:23.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 -- 172.21.15.89:0/2839900898 learned_addr learned my addr 172.21.15.89:0/2839900898 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657185b700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 0x7f656c198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657205c700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c0ff1c0 0x7f656c1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 msgr2=0x7f656c198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 0x7f656c198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c0ff1c0 msgr2=0x7f656c1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c0ff1c0 0x7f656c1988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.820+0000 7f657285d700 1 -- 172.21.15.89:0/2839900898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680095f0 con 0x7f656c1004e0 2023-08-25T02:26:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f657285d700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f656400ef30 tx=0x7f656400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564018070 con 0x7f656c1004e0 2023-08-25T02:26:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c19a8a0 con 0x7f656c1004e0 2023-08-25T02:26:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f656400f040 con 0x7f656c1004e0 2023-08-25T02:26:23.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656400cd00 con 0x7f656c1004e0 2023-08-25T02:26:23.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.821+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c1a10c0 con 0x7f656c1004e0 2023-08-25T02:26:23.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.822+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c109fa0 con 0x7f656c1004e0 2023-08-25T02:26:23.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.823+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 134688+0+0 (secure 0 0 0) 0x7f6564003a00 con 0x7f656c1004e0 2023-08-25T02:26:23.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.824+0000 7f65637fe700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 0x7f65580a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:23.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.824+0000 7f657205c700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 msgr2=0x7f65580a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:23.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.824+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f65640cb750 con 0x7f656c1004e0 2023-08-25T02:26:23.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.824+0000 7f657205c700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 0x7f65580a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:26:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:23.827+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f6564095550 con 0x7f656c1004e0 2023-08-25T02:26:23.910 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-08-25T02:26:23.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: ignoring --setuser ceph since I am not root 2023-08-25T02:26:23.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: ignoring --setgroup ceph since I am not root 2023-08-25T02:26:23.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-mgr[101583]: -- 172.21.15.121:0/3093429018 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f345da91e0 con 0x55f345d7ec00 2023-08-25T02:26:23.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:23.829+0000 7fbb26fd4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:26:23.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:23.909+0000 7fbb26fd4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:23.912 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-08-25T02:26:23.912 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: ignoring --setuser ceph since I am not root 2023-08-25T02:26:23.912 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: ignoring --setgroup ceph since I am not root 2023-08-25T02:26:23.913 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-mgr[102405]: -- 172.21.15.122:0/1326284007 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644d24c11e0 con 0x5644d2496800 2023-08-25T02:26:23.913 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:23.830+0000 7f3abc848200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:26:23.917 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:23 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-08-25T02:26:23.918 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setuser ceph since I am not root 2023-08-25T02:26:23.918 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: ignoring --setgroup ceph since I am not root 2023-08-25T02:26:23.918 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:23.833+0000 7fd27a304200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T02:26:23.918 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:23.916+0000 7fd27a304200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:24.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.025+0000 7f657205c700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 msgr2=0x7f65580a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:24.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.025+0000 7f657205c700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 0x7f65580a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:26:24.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.025+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f656c04e700 con 0x7f656c1004e0 2023-08-25T02:26:24.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.030+0000 7f65637fe700 1 -- 172.21.15.89:0/2839900898 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f65640b8070 con 0x7f656c1004e0 2023-08-25T02:26:24.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 msgr2=0x7f65580a2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:26:24.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 0x7f65580a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 msgr2=0x7f656c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f656400ef30 tx=0x7f656400c3a0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 shutdown_connections 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7f655809ffa0 0x7f65580a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656c0ff1c0 0x7f656c1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f656c1004e0 0x7f656c199c90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 --2- 172.21.15.89:0/2839900898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f656c0ffb20 0x7f656c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 >> 172.21.15.89:0/2839900898 conn(0x7f656c0fada0 msgr2=0x7f656c0fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:24.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 shutdown_connections 2023-08-25T02:26:24.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.032+0000 7f65742c0700 1 -- 172.21.15.89:0/2839900898 wait complete. 2023-08-25T02:26:24.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-08-25T02:26:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 -- 172.21.15.89:0/641698527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc101330 msgr2=0x7fbddc107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 --2- 172.21.15.89:0/641698527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc101330 0x7fbddc107be0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400b0d0 tx=0x7fbdd400b3e0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 -- 172.21.15.89:0/641698527 shutdown_connections 2023-08-25T02:26:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 --2- 172.21.15.89:0/641698527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc101330 0x7fbddc107be0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 --2- 172.21.15.89:0/641698527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc0ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 --2- 172.21.15.89:0/641698527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 0x7fbddc100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 -- 172.21.15.89:0/641698527 >> 172.21.15.89:0/641698527 conn(0x7fbddc0fad10 msgr2=0x7fbddc0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 -- 172.21.15.89:0/641698527 shutdown_connections 2023-08-25T02:26:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.169+0000 7fbde2333700 1 -- 172.21.15.89:0/641698527 wait complete. 2023-08-25T02:26:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 Processor -- start 2023-08-25T02:26:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 -- start start 2023-08-25T02:26:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 0x7fbddc196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 0x7fbddc19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc19e1d0 con 0x7fbddc0ff7a0 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc19e310 con 0x7fbddc100990 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbde2333700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc19e4a0 con 0x7fbddc101330 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbddbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbddbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48006/0 (socket says 172.21.15.89:48006) 2023-08-25T02:26:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.170+0000 7fbddbfff700 1 -- 172.21.15.89:0/3200411869 learned_addr learned my addr 172.21.15.89:0/3200411869 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 0x7fbddc19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbddb7fe700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 0x7fbddc196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 msgr2=0x7fbddc196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 0x7fbddc196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 msgr2=0x7fbddc196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 -- 172.21.15.89:0/3200411869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd4009d00 con 0x7fbddc101330 2023-08-25T02:26:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbddbfff700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde08d0700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 0x7fbddc19dc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400b770 tx=0x7fbdd4008ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd400e040 con 0x7fbddc101330 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc19e720 con 0x7fbddc101330 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdd401d930 con 0x7fbddc101330 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd401b8f0 con 0x7fbddc101330 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.171+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc19ebf0 con 0x7fbddc101330 2023-08-25T02:26:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.172+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc10afc0 con 0x7fbddc101330 2023-08-25T02:26:24.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.173+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 17) v1 ==== 134688+0+0 (secure 0 0 0) 0x7fbdd401ba50 con 0x7fbddc101330 2023-08-25T02:26:24.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.174+0000 7fbdd97fa700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 0x7fbdc40a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.174+0000 7fbddbfff700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 msgr2=0x7fbdc40a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:24.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.174+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbdd40ccd90 con 0x7fbddc101330 2023-08-25T02:26:24.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.174+0000 7fbddbfff700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 0x7fbdc40a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:26:24.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.182+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fbdd4096b90 con 0x7fbddc101330 2023-08-25T02:26:24.218 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:23.997+0000 7fbb26fd4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:26:24.218 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.090+0000 7fbb26fd4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:26:24.228 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:23.911+0000 7f3abc848200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:24.228 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.000+0000 7f3abc848200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:26:24.228 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.094+0000 7f3abc848200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:26:24.292 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.007+0000 7fd27a304200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T02:26:24.293 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.108+0000 7fd27a304200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T02:26:24.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.375+0000 7fbddbfff700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 msgr2=0x7fbdc40a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:24.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.375+0000 7fbddbfff700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 0x7fbdc40a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:26:24.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.383+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fbddc19eed0 con 0x7fbddc101330 2023-08-25T02:26:24.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.390+0000 7fbdd97fa700 1 -- 172.21.15.89:0/3200411869 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbdd409a1b0 con 0x7fbddc101330 2023-08-25T02:26:24.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.392+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 msgr2=0x7fbdc40a2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:26:24.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.392+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 0x7fbdc40a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.392+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 msgr2=0x7fbddc19dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.392+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 0x7fbddc19dc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400b770 tx=0x7fbdd4008ec0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 shutdown_connections 2023-08-25T02:26:24.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc101330 0x7fbddc19dc00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc0ff7a0 0x7fbddc196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7fbdc409ffa0 0x7fbdc40a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 --2- 172.21.15.89:0/3200411869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc100990 0x7fbddc196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 >> 172.21.15.89:0/3200411869 conn(0x7fbddc0fad10 msgr2=0x7fbddc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 shutdown_connections 2023-08-25T02:26:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.393+0000 7fbde2333700 1 -- 172.21.15.89:0/3200411869 wait complete. 2023-08-25T02:26:24.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-08-25T02:26:24.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 -- 172.21.15.89:0/3725294286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 msgr2=0x7ff07c101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 --2- 172.21.15.89:0/3725294286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c101fd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff0700099c0 tx=0x7ff070009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 -- 172.21.15.89:0/3725294286 shutdown_connections 2023-08-25T02:26:24.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 --2- 172.21.15.89:0/3725294286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c102510 0x7ff07c108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 --2- 172.21.15.89:0/3725294286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff07c100980 0x7ff07c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 --2- 172.21.15.89:0/3725294286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c101fd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 -- 172.21.15.89:0/3725294286 >> 172.21.15.89:0/3725294286 conn(0x7ff07c0fbed0 msgr2=0x7ff07c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 -- 172.21.15.89:0/3725294286 shutdown_connections 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 -- 172.21.15.89:0/3725294286 wait complete. 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.530+0000 7ff081d21700 1 Processor -- start 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 -- start start 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c100980 0x7ff07c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff07c102510 0x7ff07c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1981b0 con 0x7ff07c101b70 2023-08-25T02:26:24.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c198320 con 0x7ff07c100980 2023-08-25T02:26:24.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff081d21700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1984e0 con 0x7ff07c102510 2023-08-25T02:26:24.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48062/0 (socket says 172.21.15.89:48062) 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 -- 172.21.15.89:0/2575914556 learned_addr learned my addr 172.21.15.89:0/2575914556 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07bfff700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff07c102510 0x7ff07c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c100980 0x7ff07c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff07c102510 msgr2=0x7ff07c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff07c102510 0x7ff07c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c100980 msgr2=0x7ff07c196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:24.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c100980 0x7ff07c196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07affd700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700095f0 con 0x7ff07c101b70 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.531+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff07c100980 0x7ff07c196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff07affd700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c196cd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff070009fd0 tx=0x7ff07000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001d070 con 0x7ff07c101b70 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070005030 con 0x7ff07c101b70 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c198760 con 0x7ff07c101b70 2023-08-25T02:26:24.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070004b30 con 0x7ff07c101b70 2023-08-25T02:26:24.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.532+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c19ef80 con 0x7ff07c101b70 2023-08-25T02:26:24.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.533+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c107f60 con 0x7ff07c101b70 2023-08-25T02:26:24.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.534+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 134688+0+0 (secure 0 0 0) 0x7ff070004cd0 con 0x7ff07c101b70 2023-08-25T02:26:24.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.535+0000 7ff078ff9700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:24.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.535+0000 7ff07b7fe700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:24.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.535+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff0700cc0c0 con 0x7ff07c101b70 2023-08-25T02:26:24.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.535+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T02:26:24.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.539+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ff070095ec0 con 0x7ff07c101b70 2023-08-25T02:26:24.579 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.291+0000 7fd27a304200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:26:24.631 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.264+0000 7fbb26fd4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:26:24.639 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.271+0000 7f3abc848200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T02:26:24.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.736+0000 7ff07b7fe700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:24.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.736+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T02:26:24.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:24.777+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07c060d60 con 0x7ff06409ffa0 2023-08-25T02:26:24.944 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.630+0000 7fbb26fd4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:26:24.944 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.760+0000 7fbb26fd4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:26:24.945 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.849+0000 7fbb26fd4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:26:24.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-08-25T02:26:24.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-mon[100099]: mgrmap e17: x(active, since 7m), standbys: y, z 2023-08-25T02:26:24.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2839900898' entity='client.admin' 2023-08-25T02:26:24.945 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-mon[100099]: from='client.? ' entity='client.admin' 2023-08-25T02:26:24.954 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.637+0000 7f3abc848200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:26:24.955 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.769+0000 7f3abc848200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:26:24.955 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.858+0000 7f3abc848200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:26:24.955 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-08-25T02:26:24.956 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-mon[100457]: mgrmap e17: x(active, since 7m), standbys: y, z 2023-08-25T02:26:24.957 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2839900898' entity='client.admin' 2023-08-25T02:26:24.957 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-mon[100457]: from='client.? ' entity='client.admin' 2023-08-25T02:26:24.999 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.683+0000 7fd27a304200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T02:26:25.000 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.815+0000 7fd27a304200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T02:26:25.000 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.903+0000 7fd27a304200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T02:26:25.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1293323385' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-08-25T02:26:25.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-mon[98732]: mgrmap e17: x(active, since 7m), standbys: y, z 2023-08-25T02:26:25.000 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2839900898' entity='client.admin' 2023-08-25T02:26:25.001 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-mon[98732]: from='client.? ' entity='client.admin' 2023-08-25T02:26:25.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:25.137+0000 7ff07b7fe700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:25.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:25.137+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T02:26:25.228 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:24.953+0000 7f3abc848200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:26:25.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:24.998+0000 7fd27a304200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:26:25.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:24.943+0000 7fbb26fd4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T02:26:25.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:25.363+0000 7fbb26fd4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:26:25.728 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:25.373+0000 7f3abc848200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:26:25.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:25.420+0000 7fd27a304200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T02:26:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:25.938+0000 7ff07b7fe700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:25.938+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T02:26:26.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:26.072+0000 7fd27a304200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:26:26.329 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:26.154+0000 7fd27a304200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:26:26.468 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:26.013+0000 7fbb26fd4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:26:26.468 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:26.094+0000 7fbb26fd4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:26:26.478 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:26.023+0000 7f3abc848200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T02:26:26.478 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:26.105+0000 7f3abc848200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T02:26:27.128 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:26.863+0000 7fbb26fd4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.128 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:26.954+0000 7fbb26fd4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:26:27.139 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:26.875+0000 7f3abc848200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.139 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:26.965+0000 7f3abc848200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:26:27.190 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:26.923+0000 7fd27a304200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.191 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:27.014+0000 7fd27a304200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T02:26:27.409 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:27.126+0000 7fbb26fd4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.418 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:27.138+0000 7f3abc848200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.472 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:27.189+0000 7fd27a304200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T02:26:27.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:27.539+0000 7ff07b7fe700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1949193934 2023-08-25T02:26:27.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:27.539+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T02:26:27.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:27.407+0000 7fbb26fd4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:26:27.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:27.631+0000 7fbb26fd4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:26:27.727 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:27.417+0000 7f3abc848200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:26:27.728 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:27.640+0000 7f3abc848200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:26:27.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:27.471+0000 7fd27a304200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T02:26:27.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:27.695+0000 7fd27a304200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T02:26:28.044 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:27.781+0000 7fbb26fd4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:26:28.053 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:27.790+0000 7f3abc848200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:26:28.115 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:27.847+0000 7fd27a304200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T02:26:28.468 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:28.043+0000 7fbb26fd4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:26:28.478 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:28.052+0000 7f3abc848200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:26:28.580 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:28.113+0000 7fd27a304200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T02:26:29.061 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:28.739+0000 7fbb26fd4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:26:29.061 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:28.869+0000 7fbb26fd4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:26:29.061 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:29.060+0000 7fbb26fd4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:26:29.070 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:28.747+0000 7f3abc848200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:26:29.070 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:28.877+0000 7f3abc848200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:26:29.079 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:28.816+0000 7fd27a304200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T02:26:29.079 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:28.948+0000 7fd27a304200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T02:26:29.391 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:29.172+0000 7fbb26fd4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:29.399 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:29.069+0000 7f3abc848200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:26:29.399 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:29.182+0000 7f3abc848200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:29.480 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:29.144+0000 7fd27a304200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T02:26:29.481 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:29.259+0000 7fd27a304200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T02:26:29.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:29.390+0000 7fbb26fd4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:26:29.728 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:29.398+0000 7f3abc848200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:26:29.829 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:29.479+0000 7fd27a304200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T02:26:30.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:30.368+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 18) v1 ==== 134356+0+0 (secure 0 0 0) 0x7ff070096290 con 0x7ff07c101b70 2023-08-25T02:26:30.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:30.369+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 msgr2=0x7ff0640a2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T02:26:30.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:30.369+0000 7ff078ff9700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1949193934,v1:172.21.15.89:6801/1949193934] conn(0x7ff06409ffa0 0x7ff0640a2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:30.571 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:26:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y[101559]: 2023-08-25T02:26:30.245+0000 7fbb26fd4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:26:30.576 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:26:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z[102318]: 2023-08-25T02:26:30.255+0000 7f3abc848200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:26:30.780 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.344+0000 7fd27a304200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T02:26:30.780 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.538+0000 7fd1da531700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.538+0000 7fd1da531700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.538+0000 7fd1da531700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.538+0000 7fd1da531700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.538+0000 7fd1da531700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.540+0000 7fd1dcd36700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.540+0000 7fd1dcd36700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.781 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.540+0000 7fd1dcd36700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.782 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.540+0000 7fd1dcd36700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:30.782 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:26:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:26:30.540+0000 7fd1dcd36700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:26:31.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.374+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 19) v1 ==== 134483+0+0 (secure 0 0 0) 0x7ff0700c9bf0 con 0x7ff07c101b70 2023-08-25T02:26:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.374+0000 7ff078ff9700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff064069280 0x7ff064069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.374+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07c060d60 con 0x7ff064069280 2023-08-25T02:26:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.377+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff064069280 0x7ff064069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.379+0000 7ff07b7fe700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff064069280 0x7ff064069660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff07c1019d0 tx=0x7ff06c007400 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:31.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.439+0000 7ff078ff9700 1 -- 172.21.15.89:0/2575914556 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff07c060d60 con 0x7ff064069280 2023-08-25T02:26:31.440 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path / 2023-08-25T02:26:31.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff064069280 msgr2=0x7ff064069660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff064069280 0x7ff064069660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff07c1019d0 tx=0x7ff06c007400 comp rx=0 tx=0).stop 2023-08-25T02:26:31.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 msgr2=0x7ff07c196cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 --2- 172.21.15.89:0/2575914556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c101b70 0x7ff07c196cd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff070009fd0 tx=0x7ff07000f690 comp rx=0 tx=0).stop 2023-08-25T02:26:31.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff07bfff700 1 -- 172.21.15.89:0/2575914556 reap_dead start 2023-08-25T02:26:31.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 shutdown_connections 2023-08-25T02:26:31.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.441+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 >> 172.21.15.89:0/2575914556 conn(0x7ff07c0fbed0 msgr2=0x7ff07c1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:31.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.442+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 shutdown_connections 2023-08-25T02:26:31.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.442+0000 7ff081d21700 1 -- 172.21.15.89:0/2575914556 wait complete. 2023-08-25T02:26:31.455 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Standby manager daemon y restarted 2023-08-25T02:26:31.455 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Standby manager daemon y started 2023-08-25T02:26:31.455 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:26:31.456 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.456 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:26:31.456 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.456 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Standby manager daemon z restarted 2023-08-25T02:26:31.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Standby manager daemon z started 2023-08-25T02:26:31.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:26:31.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:26:31.457 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Active manager daemon x restarted 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Activating manager daemon x 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: osdmap e83: 12 total, 12 up, 12 in 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: mgrmap e18: x(active, starting, since 0.0114103s), standbys: y, z 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:26:31.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:26:31.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:26:31.460 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:26:31.461 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:26:31.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: Manager daemon x is now available 2023-08-25T02:26:31.463 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:26:31.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:31 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:26:31.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-08-25T02:26:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Standby manager daemon y restarted 2023-08-25T02:26:31.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Standby manager daemon y started 2023-08-25T02:26:31.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:26:31.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:26:31.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Standby manager daemon z restarted 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Standby manager daemon z started 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:26:31.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Active manager daemon x restarted 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Activating manager daemon x 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: osdmap e83: 12 total, 12 up, 12 in 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: mgrmap e18: x(active, starting, since 0.0114103s), standbys: y, z 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:26:31.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:26:31.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:26:31.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:26:31.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:26:31.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: Manager daemon x is now available 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.476 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:26:31.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:31 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:26:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Standby manager daemon y restarted 2023-08-25T02:26:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Standby manager daemon y started 2023-08-25T02:26:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-08-25T02:26:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.121:0/2694835860' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Standby manager daemon z restarted 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Standby manager daemon z started 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-08-25T02:26:31.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-08-25T02:26:31.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.? 172.21.15.122:0/1831546274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-08-25T02:26:31.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Active manager daemon x restarted 2023-08-25T02:26:31.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Activating manager daemon x 2023-08-25T02:26:31.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: osdmap e83: 12 total, 12 up, 12 in 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: mgrmap e18: x(active, starting, since 0.0114103s), standbys: y, z 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-08-25T02:26:31.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-08-25T02:26:31.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-08-25T02:26:31.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-08-25T02:26:31.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-08-25T02:26:31.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-08-25T02:26:31.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:26:31.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:26:31.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:26:31.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:26:31.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:26:31.484 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-08-25T02:26:31.485 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-08-25T02:26:31.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-08-25T02:26:31.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-08-25T02:26:31.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-08-25T02:26:31.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-08-25T02:26:31.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-08-25T02:26:31.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: Manager daemon x is now available 2023-08-25T02:26:31.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:31.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-08-25T02:26:31.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:31 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-08-25T02:26:31.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- 172.21.15.89:0/924874653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0054101b40 msgr2=0x7f0054101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 --2- 172.21.15.89:0/924874653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0054101b40 0x7f0054101fa0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00440099c0 tx=0x7f0044009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- 172.21.15.89:0/924874653 shutdown_connections 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 --2- 172.21.15.89:0/924874653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00541024e0 0x7f0054108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 --2- 172.21.15.89:0/924874653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054100950 0x7f0054100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 --2- 172.21.15.89:0/924874653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0054101b40 0x7f0054101fa0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- 172.21.15.89:0/924874653 >> 172.21.15.89:0/924874653 conn(0x7f00540fbec0 msgr2=0x7f00540fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- 172.21.15.89:0/924874653 shutdown_connections 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- 172.21.15.89:0/924874653 wait complete. 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 Processor -- start 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 -- start start 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.595+0000 7f005b43e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 0x7f0054196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054101b40 0x7f0054196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:31.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0054198130 con 0x7f00541024e0 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00541982a0 con 0x7f0054100950 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0054198460 con 0x7f0054101b40 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48292/0 (socket says 172.21.15.89:48292) 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 -- 172.21.15.89:0/1891605180 learned_addr learned my addr 172.21.15.89:0/1891605180 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00591da700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 0x7f0054196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00589d9700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054101b40 0x7f0054196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 -- 172.21.15.89:0/1891605180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054101b40 msgr2=0x7f0054196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054101b40 0x7f0054196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 -- 172.21.15.89:0/1891605180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 msgr2=0x7f0054196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 0x7f0054196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 -- 172.21.15.89:0/1891605180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440095f0 con 0x7f00541024e0 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00591da700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 0x7f0054196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:31.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f00599db700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f004c00ef30 tx=0x7f004c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:31.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c018070 con 0x7f00541024e0 2023-08-25T02:26:31.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f004c00f040 con 0x7f00541024e0 2023-08-25T02:26:31.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00541986e0 con 0x7f00541024e0 2023-08-25T02:26:31.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00cd00 con 0x7f00541024e0 2023-08-25T02:26:31.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.596+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005419ef00 con 0x7f00541024e0 2023-08-25T02:26:31.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.597+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0054107f40 con 0x7f00541024e0 2023-08-25T02:26:31.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.598+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 134483+0+0 (secure 0 0 0) 0x7f004c003a00 con 0x7f00541024e0 2023-08-25T02:26:31.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.599+0000 7f004a7fc700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 0x7f00400a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:31.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.599+0000 7f00591da700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 0x7f00400a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:31.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.599+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f004c0cb460 con 0x7f00541024e0 2023-08-25T02:26:31.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.599+0000 7f00591da700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 0x7f00400a2210 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00541019a0 tx=0x7f005000b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:31.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.603+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f004c095450 con 0x7f00541024e0 2023-08-25T02:26:31.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.833+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f0054060d60 con 0x7f004009fd50 2023-08-25T02:26:31.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.885+0000 7f004a7fc700 1 -- 172.21.15.89:0/1891605180 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f0054060d60 con 0x7f004009fd50 2023-08-25T02:26:31.886 INFO:teuthology.orchestra.run.smithi089.stdout:Retention added to path / 2023-08-25T02:26:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 msgr2=0x7f00400a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 0x7f00400a2210 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00541019a0 tx=0x7f005000b410 comp rx=0 tx=0).stop 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 msgr2=0x7f0054197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f004c00ef30 tx=0x7f004c00c3a0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 shutdown_connections 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f004009fd50 0x7f00400a2210 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0054100950 0x7f0054196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00541024e0 0x7f0054197ad0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 --2- 172.21.15.89:0/1891605180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0054101b40 0x7f0054196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:31.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 >> 172.21.15.89:0/1891605180 conn(0x7f00540fbec0 msgr2=0x7f0054105080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:31.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 shutdown_connections 2023-08-25T02:26:31.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:31.887+0000 7f005b43e700 1 -- 172.21.15.89:0/1891605180 wait complete. 2023-08-25T02:26:31.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-08-25T02:26:32.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 -- 172.21.15.89:0/2763338622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c100dd0 msgr2=0x7f978c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 --2- 172.21.15.89:0/2763338622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c100dd0 0x7f978c1011f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f978400b0a0 tx=0x7f978400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 -- 172.21.15.89:0/2763338622 shutdown_connections 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 --2- 172.21.15.89:0/2763338622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 0x7f978c107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 --2- 172.21.15.89:0/2763338622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c100dd0 0x7f978c1011f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 --2- 172.21.15.89:0/2763338622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c0ffa20 0x7f978c0ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 -- 172.21.15.89:0/2763338622 >> 172.21.15.89:0/2763338622 conn(0x7f978c0fa4f0 msgr2=0x7f978c0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 -- 172.21.15.89:0/2763338622 shutdown_connections 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 -- 172.21.15.89:0/2763338622 wait complete. 2023-08-25T02:26:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.026+0000 7f97931b1700 1 Processor -- start 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 -- start start 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 0x7f978c195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c100dd0 0x7f978c196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c197150 con 0x7f978c0ffa20 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1972c0 con 0x7f978c100dd0 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c197480 con 0x7f978c1003e0 2023-08-25T02:26:32.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48302/0 (socket says 172.21.15.89:48302) 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 -- 172.21.15.89:0/1313736161 learned_addr learned my addr 172.21.15.89:0/1313736161 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f978bfff700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 0x7f978c195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f979174e700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c100dd0 0x7f978c196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 -- 172.21.15.89:0/1313736161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 msgr2=0x7f978c195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 0x7f978c195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 -- 172.21.15.89:0/1313736161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c100dd0 msgr2=0x7f978c196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c100dd0 0x7f978c196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 -- 172.21.15.89:0/1313736161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009d00 con 0x7f978c0ffa20 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9790f4d700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9784006070 tx=0x7f9784004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400e040 con 0x7f978c0ffa20 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f978401db50 con 0x7f978c0ffa20 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c197700 con 0x7f978c0ffa20 2023-08-25T02:26:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.027+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978401bbb0 con 0x7f978c0ffa20 2023-08-25T02:26:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.028+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c19df20 con 0x7f978c0ffa20 2023-08-25T02:26:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.029+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c106e30 con 0x7f978c0ffa20 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.030+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 134483+0+0 (secure 0 0 0) 0x7f978401dcc0 con 0x7f978c0ffa20 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.030+0000 7f9789ffb700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 0x7f97780a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.030+0000 7f978bfff700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 0x7f97780a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.030+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f97840cd0b0 con 0x7f978c0ffa20 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.031+0000 7f978bfff700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 0x7f97780a2290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9774007c60 tx=0x7f97740073d0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:32.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.034+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f9784097120 con 0x7f978c0ffa20 2023-08-25T02:26:32.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.268+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f978c060d60 con 0x7f977809fdd0 2023-08-25T02:26:32.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.276+0000 7f9789ffb700 1 -- 172.21.15.89:0/1313736161 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f978c060d60 con 0x7f977809fdd0 2023-08-25T02:26:32.278 INFO:teuthology.orchestra.run.smithi089.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-08-25T00:00:00", "created": "2023-08-25T02:26:31", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-08-25T02:26:32.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 msgr2=0x7f97780a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 0x7f97780a2290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9774007c60 tx=0x7f97740073d0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 msgr2=0x7f978c195730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9784006070 tx=0x7f9784004d00 comp rx=0 tx=0).stop 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 shutdown_connections 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f977809fdd0 0x7f97780a2290 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f978c100dd0 0x7f978c196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f978c0ffa20 0x7f978c195730 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 --2- 172.21.15.89:0/1313736161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f978c1003e0 0x7f978c195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 >> 172.21.15.89:0/1313736161 conn(0x7f978c0fa4f0 msgr2=0x7f978c103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 shutdown_connections 2023-08-25T02:26:32.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.279+0000 7f97931b1700 1 -- 172.21.15.89:0/1313736161 wait complete. 2023-08-25T02:26:32.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-08-25T02:26:32.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3411465936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e01013a0 msgr2=0x7fe9e0103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3411465936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e01013a0 0x7fe9e0103790 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800b0a0 tx=0x7fe9d800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3411465936 shutdown_connections 2023-08-25T02:26:32.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3411465936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e0106600 0x7fe9e0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3411465936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e01013a0 0x7fe9e0103790 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3411465936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0103cd0 0x7fe9e01060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3411465936 >> 172.21.15.89:0/3411465936 conn(0x7fe9e00fad70 msgr2=0x7fe9e00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:32.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3411465936 shutdown_connections 2023-08-25T02:26:32.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3411465936 wait complete. 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 Processor -- start 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 -- start start 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e01013a0 0x7fe9e01967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.421+0000 7fe9e87d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0106600 0x7fe9e0197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01981c0 con 0x7fe9e0103cd0 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0198330 con 0x7fe9e01013a0 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01984f0 con 0x7fe9e0106600 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48318/0 (socket says 172.21.15.89:48318) 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 -- 172.21.15.89:0/3543645185 learned_addr learned my addr 172.21.15.89:0/3543645185 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e656e700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e01013a0 0x7fe9e01967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e6d6f700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0106600 0x7fe9e0197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 -- 172.21.15.89:0/3543645185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0106600 msgr2=0x7fe9e0197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0106600 0x7fe9e0197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 -- 172.21.15.89:0/3543645185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e01013a0 msgr2=0x7fe9e01967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e01013a0 0x7fe9e01967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 -- 172.21.15.89:0/3543645185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d8009d00 con 0x7fe9e0103cd0 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e5d6d700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e9a0 tx=0x7fe9d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0009790 con 0x7fe9e0103cd0 2023-08-25T02:26:32.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9d000ce80 con 0x7fe9e0103cd0 2023-08-25T02:26:32.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e01987d0 con 0x7fe9e0103cd0 2023-08-25T02:26:32.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0018680 con 0x7fe9e0103cd0 2023-08-25T02:26:32.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.422+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e019ef90 con 0x7fe9e0103cd0 2023-08-25T02:26:32.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.423+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e00fc8f0 con 0x7fe9e0103cd0 2023-08-25T02:26:32.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.424+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 134483+0+0 (secure 0 0 0) 0x7fe9d0010430 con 0x7fe9e0103cd0 2023-08-25T02:26:32.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.425+0000 7fe9d77fe700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 0x7fe9cc0a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:32.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.425+0000 7fe9e656e700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 0x7fe9cc0a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:32.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.425+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe9d0014070 con 0x7fe9e0103cd0 2023-08-25T02:26:32.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.425+0000 7fe9e656e700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 0x7fe9cc0a2210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8007a60 tx=0x7fe9d800b700 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:32.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.428+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fe9d0095470 con 0x7fe9e0103cd0 2023-08-25T02:26:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: [25/Aug/2023:02:26:30] ENGINE Bus STARTING 2023-08-25T02:26:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: [25/Aug/2023:02:26:30] ENGINE Serving on https://172.21.15.89:7150 2023-08-25T02:26:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: [25/Aug/2023:02:26:30] ENGINE Serving on http://172.21.15.89:8765 2023-08-25T02:26:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: [25/Aug/2023:02:26:30] ENGINE Bus STARTED 2023-08-25T02:26:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: mgrmap e19: x(active, since 1.01748s), standbys: y, z 2023-08-25T02:26:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:32 smithi089 ceph-mon[98732]: pgmap v3: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.662+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe9e002cbe0 con 0x7fe9cc09fd50 2023-08-25T02:26:32.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.670+0000 7fe9d77fe700 1 -- 172.21.15.89:0/3543645185 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fe9e002cbe0 con 0x7fe9cc09fd50 2023-08-25T02:26:32.671 INFO:teuthology.orchestra.run.smithi089.stdout:/ 1M 6M3h 2023-08-25T02:26:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 msgr2=0x7fe9cc0a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 0x7fe9cc0a2210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8007a60 tx=0x7fe9d800b700 comp rx=0 tx=0).stop 2023-08-25T02:26:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 msgr2=0x7fe9e0196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e9a0 tx=0x7fe9d000ecb0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 shutdown_connections 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe9cc09fd50 0x7fe9cc0a2210 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9e01013a0 0x7fe9e01967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9e0106600 0x7fe9e0197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 --2- 172.21.15.89:0/3543645185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e0103cd0 0x7fe9e0196ce0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 >> 172.21.15.89:0/3543645185 conn(0x7fe9e00fad70 msgr2=0x7fe9e0102010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 shutdown_connections 2023-08-25T02:26:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:26:32.672+0000 7fe9e87d2700 1 -- 172.21.15.89:0/3543645185 wait complete. 2023-08-25T02:26:32.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-08-25T02:26:32.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: [25/Aug/2023:02:26:30] ENGINE Bus STARTING 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: [25/Aug/2023:02:26:30] ENGINE Serving on https://172.21.15.89:7150 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: [25/Aug/2023:02:26:30] ENGINE Serving on http://172.21.15.89:8765 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: [25/Aug/2023:02:26:30] ENGINE Bus STARTED 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: mgrmap e19: x(active, since 1.01748s), standbys: y, z 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:32.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:32 smithi121 ceph-mon[100099]: pgmap v3: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:32.720 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-08-25T02:26:32.863 INFO:teuthology.run_tasks:Running task background_exec... 2023-08-25T02:26:32.899 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:26:32.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-08-25T02:26:32.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: [25/Aug/2023:02:26:30] ENGINE Bus STARTING 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: [25/Aug/2023:02:26:30] ENGINE Serving on https://172.21.15.89:7150 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: [25/Aug/2023:02:26:30] ENGINE Serving on http://172.21.15.89:8765 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: [25/Aug/2023:02:26:30] ENGINE Bus STARTED 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: mgrmap e19: x(active, since 1.01748s), standbys: y, z 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:32.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:32 smithi122 ceph-mon[100457]: pgmap v3: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:32.902 INFO:teuthology.run_tasks:Running task exec... 2023-08-25T02:26:32.912 INFO:teuthology.task.exec:Executing custom commands... 2023-08-25T02:26:32.912 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:26:32.912 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2023-08-25T02:26:32.952 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:32.962 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem Size Used Avail Use% Mounted on 2023-08-25T02:26:32.962 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:3300,172.21.15.121:3300,172.21.15.122:3300:/ 340G 0 340G 0% /tmp/cephfs 2023-08-25T02:26:32.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2023-08-25T02:26:33.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2023-08-25T02:26:33.099 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2023-08-25T02:26:33.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: pgmap v4: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:33 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: pgmap v4: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:33 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: pgmap v4: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:33 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: mgrmap e20: x(active, since 3s), standbys: y, z 2023-08-25T02:26:34.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:34.723 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:34 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:34.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: mgrmap e20: x(active, since 3s), standbys: y, z 2023-08-25T02:26:34.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:34.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:34 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: mgrmap e20: x(active, since 3s), standbys: y, z 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:26:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:34.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:34 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:35.714 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:26:35.715 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:26:35.715 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:26:35.715 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.715 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:26:35.716 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.716 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:26:35.716 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:26:35.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:26:35.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: pgmap v5: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:35.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.717 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.718 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: mgrmap e21: x(active, since 4s), standbys: y, z 2023-08-25T02:26:35.718 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.718 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.718 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.718 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:35 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:26:35.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:26:35.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:26:35.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: pgmap v5: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:35.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: mgrmap e21: x(active, since 4s), standbys: y, z 2023-08-25T02:26:35.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:35 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:26:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:26:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:26:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:26:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi089:/etc/ceph/ceph.conf 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi121:/etc/ceph/ceph.conf 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi122:/etc/ceph/ceph.conf 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: pgmap v5: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.conf 2023-08-25T02:26:35.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: mgrmap e21: x(active, since 4s), standbys: y, z 2023-08-25T02:26:35.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:35 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:35.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2023-08-25T02:26:35.890 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2023-08-25T02:26:35.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2023-08-25T02:26:35.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2023-08-25T02:26:36.051 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/: 2023-08-25T02:26:36.051 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:36.052 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:36.052 INFO:teuthology.orchestra.run.smithi089.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 .. 2023-08-25T02:26:36.052 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 data 2023-08-25T02:26:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:36.053 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data: 2023-08-25T02:26:36.053 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:36.053 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:36.054 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 .. 2023-08-25T02:26:36.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2023-08-25T02:26:36.140 INFO:teuthology.orchestra.run.smithi089.stdout:[Service] 2023-08-25T02:26:36.140 INFO:teuthology.orchestra.run.smithi089.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2023-08-25T02:26:36.140 INFO:teuthology.orchestra.run.smithi089.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2023-08-25T02:26:36.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2023-08-25T02:26:36.233 INFO:teuthology.orchestra.run.smithi089.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2023-08-25T02:26:36.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:36.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:36.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:36.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: Updating smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/config/ceph.client.admin.keyring 2023-08-25T02:26:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:26:36.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:26:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:37 smithi089 ceph-mon[98732]: pgmap v6: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:37 smithi121 ceph-mon[100099]: pgmap v6: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:37 smithi122 ceph-mon[100457]: pgmap v6: 97 pgs: 97 active+clean; 969 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-08-25T02:26:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:39 smithi089 ceph-mon[98732]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 151 KiB/s wr, 70 op/s 2023-08-25T02:26:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:39 smithi121 ceph-mon[100099]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 151 KiB/s wr, 70 op/s 2023-08-25T02:26:39.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:39 smithi122 ceph-mon[100457]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 151 KiB/s wr, 70 op/s 2023-08-25T02:26:40.140 INFO:teuthology.orchestra.run.smithi089.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2023-08-25T02:26:40.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2023-08-25T02:26:40.188 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/: 2023-08-25T02:26:40.188 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 .. 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 data 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data: 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:total 30 2023-08-25T02:26:40.189 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 . 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 .. 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 base 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Aug 25 02:26 global 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 log 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_commit_ts 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_dynshmem 2023-08-25T02:26:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4269 Aug 25 02:26 pg_hba.conf 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Aug 25 02:26 pg_ident.conf 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 pg_logical 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 pg_multixact 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 pg_notify 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_replslot 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_serial 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_snapshots 2023-08-25T02:26:40.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_stat 2023-08-25T02:26:40.192 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_stat_tmp 2023-08-25T02:26:40.192 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 pg_subtrans 2023-08-25T02:26:40.192 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_tblspc 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 pg_twophase 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 PG_VERSION 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 pg_wal 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 pg_xact 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Aug 25 02:26 postgresql.auto.conf 2023-08-25T02:26:40.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 22989 Aug 25 02:26 postgresql.conf 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/base: 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 . 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 1 2023-08-25T02:26:40.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 12346 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 12347 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/base/1: 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:total 7201 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 . 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 .. 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 112 2023-08-25T02:26:40.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 113 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Aug 25 02:26 12182 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12182_fsm 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12182_vm 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12184 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12186 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187 2023-08-25T02:26:40.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12187_fsm 2023-08-25T02:26:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187_vm 2023-08-25T02:26:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12189 2023-08-25T02:26:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12191 2023-08-25T02:26:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192 2023-08-25T02:26:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12192_fsm 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192_vm 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12194 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12196 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12197_fsm 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197_vm 2023-08-25T02:26:40.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12199 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12201 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12202_fsm 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202_vm 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12204 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12206 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207 2023-08-25T02:26:40.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12207_fsm 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207_vm 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12209 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12211 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12212 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12214 2023-08-25T02:26:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12216 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Aug 25 02:26 1247 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1247_fsm 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1247_vm 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Aug 25 02:26 1249 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1249_fsm 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1249_vm 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Aug 25 02:26 1255 2023-08-25T02:26:40.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1255_fsm 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1255_vm 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Aug 25 02:26 1259 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1259_fsm 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1259_vm 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417_vm 2023-08-25T02:26:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418_vm 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 174 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 175 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2187 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224 2023-08-25T02:26:40.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224_vm 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328_vm 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336_vm 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2337 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2600 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2600_fsm 2023-08-25T02:26:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2600_vm 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2601_fsm 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601_vm 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2602 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2602_fsm 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2602_vm 2023-08-25T02:26:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2603 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2603_fsm 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2603_vm 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604_vm 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2605 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2605_fsm 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2605_vm 2023-08-25T02:26:40.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2606_fsm 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606_vm 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607_fsm 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2607_vm 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Aug 25 02:26 2608 2023-08-25T02:26:40.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2608_fsm 2023-08-25T02:26:40.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2608_vm 2023-08-25T02:26:40.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Aug 25 02:26 2609 2023-08-25T02:26:40.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2609_fsm 2023-08-25T02:26:40.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2609_vm 2023-08-25T02:26:40.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610_fsm 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2610_vm 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611_vm 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2612_fsm 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612_vm 2023-08-25T02:26:40.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613_vm 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2615_fsm 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615_vm 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616_fsm 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2616_vm 2023-08-25T02:26:40.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Aug 25 02:26 2617 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2617_fsm 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2617_vm 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Aug 25 02:26 2618 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2618_fsm 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2618_vm 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Aug 25 02:26 2619 2023-08-25T02:26:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2619_fsm 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2619_vm 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620_vm 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2650 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2651 2023-08-25T02:26:40.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2652 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2653 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2654 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2655 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2656 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2657 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Aug 25 02:26 2658 2023-08-25T02:26:40.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2659 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2660 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2661 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2662 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2663 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2664 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2665 2023-08-25T02:26:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2666 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2667 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2668 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2669 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2670 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Aug 25 02:26 2673 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Aug 25 02:26 2674 2023-08-25T02:26:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Aug 25 02:26 2675 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2678 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2679 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2680 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2681 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2682 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2683 2023-08-25T02:26:40.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2684 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2685 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2686 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2687 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2688 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2689 2023-08-25T02:26:40.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2690 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Aug 25 02:26 2691 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2692 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2693 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2696 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2699 2023-08-25T02:26:40.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2701 2023-08-25T02:26:40.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2702 2023-08-25T02:26:40.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2703 2023-08-25T02:26:40.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2704 2023-08-25T02:26:40.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2753 2023-08-25T02:26:40.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2753_fsm 2023-08-25T02:26:40.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2753_vm 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2754 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2755 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2756 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2757 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830_vm 2023-08-25T02:26:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2831 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832_vm 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2833 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834_vm 2023-08-25T02:26:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2835 2023-08-25T02:26:40.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836 2023-08-25T02:26:40.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836_vm 2023-08-25T02:26:40.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2837 2023-08-25T02:26:40.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Aug 25 02:26 2838 2023-08-25T02:26:40.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2838_fsm 2023-08-25T02:26:40.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2838_vm 2023-08-25T02:26:40.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2839 2023-08-25T02:26:40.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840 2023-08-25T02:26:40.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840_fsm 2023-08-25T02:26:40.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2840_vm 2023-08-25T02:26:40.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2841 2023-08-25T02:26:40.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995 2023-08-25T02:26:40.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995_vm 2023-08-25T02:26:40.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2996 2023-08-25T02:26:40.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079 2023-08-25T02:26:40.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3079_fsm 2023-08-25T02:26:40.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079_vm 2023-08-25T02:26:40.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3080 2023-08-25T02:26:40.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3081 2023-08-25T02:26:40.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3085 2023-08-25T02:26:40.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118 2023-08-25T02:26:40.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118_vm 2023-08-25T02:26:40.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3119 2023-08-25T02:26:40.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3164 2023-08-25T02:26:40.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256 2023-08-25T02:26:40.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256_vm 2023-08-25T02:26:40.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3257 2023-08-25T02:26:40.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3258 2023-08-25T02:26:40.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350 2023-08-25T02:26:40.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350_vm 2023-08-25T02:26:40.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3351 2023-08-25T02:26:40.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3379 2023-08-25T02:26:40.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3380 2023-08-25T02:26:40.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381 2023-08-25T02:26:40.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381_vm 2023-08-25T02:26:40.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3394 2023-08-25T02:26:40.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3394_fsm 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3394_vm 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3395 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439_vm 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3440 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 3455 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3456 2023-08-25T02:26:40.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3456_fsm 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3456_vm 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466_vm 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3467 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3468 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501 2023-08-25T02:26:40.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501_vm 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3502 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3503 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3534 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3541_fsm 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541_vm 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3542 2023-08-25T02:26:40.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3574 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3575 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576_vm 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596_vm 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3597 2023-08-25T02:26:40.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598_vm 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3599 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3600_fsm 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600_vm 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3601_fsm 2023-08-25T02:26:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601_vm 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3602_fsm 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602_vm 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3603 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3603_fsm 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3603_vm 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3604 2023-08-25T02:26:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3605 2023-08-25T02:26:40.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3606 2023-08-25T02:26:40.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3607 2023-08-25T02:26:40.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3608 2023-08-25T02:26:40.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 3609 2023-08-25T02:26:40.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3712 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3764_fsm 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764_vm 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3766 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3767 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3997 2023-08-25T02:26:40.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 5002 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 548 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 549 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102_vm 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104_vm 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106 2023-08-25T02:26:40.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106_vm 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6110 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6111 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6112 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6113 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6117 2023-08-25T02:26:40.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826_vm 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 827 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 828 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Aug 25 02:26 pg_filenode.map 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 PG_VERSION 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/base/12346: 2023-08-25T02:26:40.245 INFO:teuthology.orchestra.run.smithi089.stdout:total 7201 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 . 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 .. 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 112 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 113 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Aug 25 02:26 12182 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12182_fsm 2023-08-25T02:26:40.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12182_vm 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12184 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12186 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12187_fsm 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187_vm 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12189 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12191 2023-08-25T02:26:40.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12192_fsm 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192_vm 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12194 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12196 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12197_fsm 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197_vm 2023-08-25T02:26:40.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12199 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12201 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12202_fsm 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202_vm 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12204 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12206 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207 2023-08-25T02:26:40.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12207_fsm 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207_vm 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12209 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12211 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12212 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12214 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12216 2023-08-25T02:26:40.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Aug 25 02:26 1247 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1247_fsm 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1247_vm 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Aug 25 02:26 1249 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1249_fsm 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1249_vm 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Aug 25 02:26 1255 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1255_fsm 2023-08-25T02:26:40.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1255_vm 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Aug 25 02:26 1259 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1259_fsm 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1259_vm 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417_vm 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418 2023-08-25T02:26:40.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418_vm 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 174 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 175 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2187 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224_vm 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328 2023-08-25T02:26:40.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328_vm 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336_vm 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2337 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2600 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2600_fsm 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2600_vm 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601 2023-08-25T02:26:40.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2601_fsm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601_vm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2602 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2602_fsm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2602_vm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2603 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2603_fsm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2603_vm 2023-08-25T02:26:40.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604_vm 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2605 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2605_fsm 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2605_vm 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2606_fsm 2023-08-25T02:26:40.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606_vm 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607_fsm 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2607_vm 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Aug 25 02:26 2608 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2608_fsm 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2608_vm 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Aug 25 02:26 2609 2023-08-25T02:26:40.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2609_fsm 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2609_vm 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610_fsm 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2610_vm 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611_vm 2023-08-25T02:26:40.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2612_fsm 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612_vm 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613_vm 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2615_fsm 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615_vm 2023-08-25T02:26:40.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616_fsm 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2616_vm 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Aug 25 02:26 2617 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2617_fsm 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2617_vm 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Aug 25 02:26 2618 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2618_fsm 2023-08-25T02:26:40.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2618_vm 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Aug 25 02:26 2619 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2619_fsm 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2619_vm 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620_vm 2023-08-25T02:26:40.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2650 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2651 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2652 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2653 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2654 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2655 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2656 2023-08-25T02:26:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2657 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Aug 25 02:26 2658 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2659 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2660 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2661 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2662 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2663 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2664 2023-08-25T02:26:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2665 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2666 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2667 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2668 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2669 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2670 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Aug 25 02:26 2673 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Aug 25 02:26 2674 2023-08-25T02:26:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Aug 25 02:26 2675 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2678 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2679 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2680 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2681 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2682 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2683 2023-08-25T02:26:40.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2684 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2685 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2686 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2687 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2688 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2689 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2690 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Aug 25 02:26 2691 2023-08-25T02:26:40.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2692 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2693 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2696 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2699 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2701 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2702 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2703 2023-08-25T02:26:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2704 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2753 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2753_fsm 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2753_vm 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2754 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2755 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2756 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2757 2023-08-25T02:26:40.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830_vm 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2831 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832_vm 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2833 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834 2023-08-25T02:26:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834_vm 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2835 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836_vm 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2837 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Aug 25 02:26 2838 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2838_fsm 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2838_vm 2023-08-25T02:26:40.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2839 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840_fsm 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2840_vm 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2841 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995_vm 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2996 2023-08-25T02:26:40.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3079_fsm 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079_vm 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3080 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3081 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3085 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118_vm 2023-08-25T02:26:40.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3119 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3164 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256_vm 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3257 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3258 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350 2023-08-25T02:26:40.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350_vm 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3351 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3379 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3380 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381_vm 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3394 2023-08-25T02:26:40.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3394_fsm 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3394_vm 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3395 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439_vm 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3440 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 3455 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3456 2023-08-25T02:26:40.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3456_fsm 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3456_vm 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466_vm 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3467 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3468 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501 2023-08-25T02:26:40.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501_vm 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3502 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3503 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3534 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3541_fsm 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541_vm 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3542 2023-08-25T02:26:40.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3574 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3575 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576_vm 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596_vm 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3597 2023-08-25T02:26:40.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598_vm 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3599 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3600_fsm 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600_vm 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3601_fsm 2023-08-25T02:26:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601_vm 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3602_fsm 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602_vm 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3603 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3603_fsm 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3603_vm 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3604 2023-08-25T02:26:40.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3605 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3606 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3607 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3608 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 3609 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3712 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764 2023-08-25T02:26:40.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3764_fsm 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764_vm 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3766 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3767 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3997 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 5002 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 548 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 549 2023-08-25T02:26:40.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102_vm 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104_vm 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106_vm 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6110 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6111 2023-08-25T02:26:40.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6112 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6113 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6117 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826_vm 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 827 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 828 2023-08-25T02:26:40.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Aug 25 02:26 pg_filenode.map 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 PG_VERSION 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/base/12347: 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:total 7201 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Aug 25 02:26 . 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 .. 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 112 2023-08-25T02:26:40.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 113 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Aug 25 02:26 12182 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12182_fsm 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12182_vm 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12184 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12186 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12187_fsm 2023-08-25T02:26:40.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12187_vm 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12189 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12191 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12192_fsm 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12192_vm 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12194 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12196 2023-08-25T02:26:40.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12197_fsm 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12197_vm 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12199 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12201 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12202_fsm 2023-08-25T02:26:40.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12202_vm 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12204 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12206 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 12207_fsm 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12207_vm 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12209 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12211 2023-08-25T02:26:40.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12212 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 12214 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 12216 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Aug 25 02:26 1247 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1247_fsm 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1247_vm 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Aug 25 02:26 1249 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1249_fsm 2023-08-25T02:26:40.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1249_vm 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Aug 25 02:26 1255 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1255_fsm 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1255_vm 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Aug 25 02:26 1259 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1259_fsm 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1259_vm 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417 2023-08-25T02:26:40.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1417_vm 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 1418_vm 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 174 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 175 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2187 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2224_vm 2023-08-25T02:26:40.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2328_vm 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2336_vm 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2337 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2600 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2600_fsm 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2600_vm 2023-08-25T02:26:40.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2601_fsm 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2601_vm 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2602 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2602_fsm 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2602_vm 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2603 2023-08-25T02:26:40.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2603_fsm 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2603_vm 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2604_vm 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2605 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2605_fsm 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2605_vm 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606 2023-08-25T02:26:40.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2606_fsm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2606_vm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2607_fsm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2607_vm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Aug 25 02:26 2608 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2608_fsm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2608_vm 2023-08-25T02:26:40.296 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Aug 25 02:26 2609 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2609_fsm 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2609_vm 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2610_fsm 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2610_vm 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2611_vm 2023-08-25T02:26:40.297 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2612_fsm 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2612_vm 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2613_vm 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2615_fsm 2023-08-25T02:26:40.298 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2615_vm 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2616_fsm 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2616_vm 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Aug 25 02:26 2617 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2617_fsm 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2617_vm 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Aug 25 02:26 2618 2023-08-25T02:26:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2618_fsm 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2618_vm 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Aug 25 02:26 2619 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2619_fsm 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2619_vm 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2620_vm 2023-08-25T02:26:40.300 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2650 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2651 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2652 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2653 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2654 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2655 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2656 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2657 2023-08-25T02:26:40.301 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Aug 25 02:26 2658 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2659 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2660 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2661 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2662 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2663 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2664 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2665 2023-08-25T02:26:40.302 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2666 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2667 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2668 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2669 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2670 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Aug 25 02:26 2673 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Aug 25 02:26 2674 2023-08-25T02:26:40.303 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Aug 25 02:26 2675 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2678 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2679 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2680 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2681 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2682 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2683 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2684 2023-08-25T02:26:40.304 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2685 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2686 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2687 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2688 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Aug 25 02:26 2689 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Aug 25 02:26 2690 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Aug 25 02:26 2691 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2692 2023-08-25T02:26:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2693 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2696 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2699 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2701 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2702 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2703 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 2704 2023-08-25T02:26:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2753 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2753_fsm 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2753_vm 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2754 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2755 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2756 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 2757 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830 2023-08-25T02:26:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2830_vm 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2831 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2832_vm 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2833 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2834_vm 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2835 2023-08-25T02:26:40.308 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2836_vm 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2837 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Aug 25 02:26 2838 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2838_fsm 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2838_vm 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2839 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840 2023-08-25T02:26:40.309 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2840_fsm 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2840_vm 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2841 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2995_vm 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2996 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079 2023-08-25T02:26:40.310 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3079_fsm 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3079_vm 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3080 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3081 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3085 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3118_vm 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3119 2023-08-25T02:26:40.311 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3164 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3256_vm 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3257 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3258 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3350_vm 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3351 2023-08-25T02:26:40.312 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3379 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3380 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3381_vm 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3394 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3394_fsm 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3394_vm 2023-08-25T02:26:40.313 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3395 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3439_vm 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3440 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Aug 25 02:26 3455 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3456 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3456_fsm 2023-08-25T02:26:40.314 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3456_vm 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3466_vm 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3467 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3468 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3501_vm 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3502 2023-08-25T02:26:40.315 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3503 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3534 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3541_fsm 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3541_vm 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3542 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3574 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3575 2023-08-25T02:26:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3576_vm 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3596_vm 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3597 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3598_vm 2023-08-25T02:26:40.317 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3599 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3600_fsm 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3600_vm 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3601_fsm 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3601_vm 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602 2023-08-25T02:26:40.318 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3602_fsm 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3602_vm 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3603 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3603_fsm 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3603_vm 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3604 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3605 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3606 2023-08-25T02:26:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3607 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3608 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Aug 25 02:26 3609 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3712 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 3764_fsm 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3764_vm 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3766 2023-08-25T02:26:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 3767 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3997 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 5002 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 548 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 549 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6102_vm 2023-08-25T02:26:40.321 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6104_vm 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6106_vm 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6110 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6111 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6112 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6113 2023-08-25T02:26:40.322 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6117 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 826_vm 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 827 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 828 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Aug 25 02:26 pg_filenode.map 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 PG_VERSION 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.323 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/global: 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:total 553 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Aug 25 02:26 . 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1136 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1136_fsm 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1136_vm 2023-08-25T02:26:40.324 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 1137 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1213 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1213_fsm 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1213_vm 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1214 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1214_fsm 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1214_vm 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 1232 2023-08-25T02:26:40.325 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 1233 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1260 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1260_fsm 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1260_vm 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1261 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1261_fsm 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1261_vm 2023-08-25T02:26:40.326 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1262 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 1262_fsm 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 1262_vm 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2396 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Aug 25 02:26 2396_fsm 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2396_vm 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2397 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2671 2023-08-25T02:26:40.327 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2672 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2676 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2677 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2694 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2695 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2697 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Aug 25 02:26 2698 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2846 2023-08-25T02:26:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2846_vm 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2847 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2964 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2964_vm 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2965 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2966 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 2966_vm 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 2967 2023-08-25T02:26:40.329 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3592 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 3592_vm 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 3593 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 4060 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 4060_vm 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 4061 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6000 2023-08-25T02:26:40.330 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6000_vm 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6001 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6002 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6100 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 6100_vm 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6114 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 6115 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 pg_control 2023-08-25T02:26:40.331 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Aug 25 02:26 pg_filenode.map 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/log: 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2023-08-25T02:26:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.333 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_logical: 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:total 1 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 . 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 mappings 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Aug 25 02:26 replorigin_checkpoint 2023-08-25T02:26:40.334 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 snapshots 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 .. 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2023-08-25T02:26:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Aug 25 02:26 .. 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 . 2023-08-25T02:26:40.336 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 members 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 offsets 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:total 8 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 .. 2023-08-25T02:26:40.337 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 0000 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:total 8 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 .. 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 0000 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.338 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_notify: 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:total 8 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 0000 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.339 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_serial: 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.340 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_stat: 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.341 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.342 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:total 8 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 0000 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.343 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.344 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_wal: 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:total 16384 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 . 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Aug 25 02:26 000000010000000000000001 2023-08-25T02:26:40.345 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 archive_status 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:total 0 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Aug 25 02:26 . 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Aug 25 02:26 .. 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/cephfs/postgres/data/pg_xact: 2023-08-25T02:26:40.346 INFO:teuthology.orchestra.run.smithi089.stdout:total 8 2023-08-25T02:26:40.347 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Aug 25 02:26 . 2023-08-25T02:26:40.347 INFO:teuthology.orchestra.run.smithi089.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Aug 25 02:26 .. 2023-08-25T02:26:40.347 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Aug 25 02:26 0000 2023-08-25T02:26:40.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2023-08-25T02:26:40.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i' 2023-08-25T02:26:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2023-08-25T02:26:40.529 INFO:teuthology.orchestra.run.smithi089.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2023-08-25T02:26:40.531 INFO:teuthology.orchestra.run.smithi089.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2023-08-25T02:26:40.534 INFO:teuthology.orchestra.run.smithi089.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2023-08-25T02:26:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:creating tables... 2023-08-25T02:26:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:100000 of 50000000 tuples (0%) done (elapsed 0.06 s, remaining 28.48 s) 2023-08-25T02:26:40.924 INFO:teuthology.orchestra.run.smithi089.stderr:200000 of 50000000 tuples (0%) done (elapsed 0.22 s, remaining 55.28 s) 2023-08-25T02:26:41.115 INFO:teuthology.orchestra.run.smithi089.stderr:300000 of 50000000 tuples (0%) done (elapsed 0.41 s, remaining 68.41 s) 2023-08-25T02:26:41.177 INFO:teuthology.orchestra.run.smithi089.stderr:400000 of 50000000 tuples (0%) done (elapsed 0.47 s, remaining 58.90 s) 2023-08-25T02:26:41.350 INFO:teuthology.orchestra.run.smithi089.stderr:500000 of 50000000 tuples (1%) done (elapsed 0.65 s, remaining 64.18 s) 2023-08-25T02:26:41.453 INFO:teuthology.orchestra.run.smithi089.stderr:600000 of 50000000 tuples (1%) done (elapsed 0.75 s, remaining 61.84 s) 2023-08-25T02:26:41.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:41 smithi121 ceph-mon[100099]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 117 KiB/s wr, 54 op/s 2023-08-25T02:26:41.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:41 smithi122 ceph-mon[100457]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 117 KiB/s wr, 54 op/s 2023-08-25T02:26:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:41 smithi089 ceph-mon[98732]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 117 KiB/s wr, 54 op/s 2023-08-25T02:26:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:700000 of 50000000 tuples (1%) done (elapsed 0.92 s, remaining 65.01 s) 2023-08-25T02:26:41.796 INFO:teuthology.orchestra.run.smithi089.stderr:800000 of 50000000 tuples (1%) done (elapsed 1.09 s, remaining 67.27 s) 2023-08-25T02:26:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:900000 of 50000000 tuples (1%) done (elapsed 1.26 s, remaining 68.52 s) 2023-08-25T02:26:42.037 INFO:teuthology.orchestra.run.smithi089.stderr:1000000 of 50000000 tuples (2%) done (elapsed 1.33 s, remaining 65.41 s) 2023-08-25T02:26:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:1100000 of 50000000 tuples (2%) done (elapsed 1.52 s, remaining 67.73 s) 2023-08-25T02:26:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:1200000 of 50000000 tuples (2%) done (elapsed 1.63 s, remaining 66.34 s) 2023-08-25T02:26:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:1300000 of 50000000 tuples (2%) done (elapsed 1.80 s, remaining 67.39 s) 2023-08-25T02:26:42.678 INFO:teuthology.orchestra.run.smithi089.stderr:1400000 of 50000000 tuples (2%) done (elapsed 1.98 s, remaining 68.58 s) 2023-08-25T02:26:42.780 INFO:teuthology.orchestra.run.smithi089.stderr:1500000 of 50000000 tuples (3%) done (elapsed 2.08 s, remaining 67.19 s) 2023-08-25T02:26:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:1600000 of 50000000 tuples (3%) done (elapsed 2.24 s, remaining 67.74 s) 2023-08-25T02:26:43.105 INFO:teuthology.orchestra.run.smithi089.stderr:1700000 of 50000000 tuples (3%) done (elapsed 2.40 s, remaining 68.26 s) 2023-08-25T02:26:43.184 INFO:teuthology.orchestra.run.smithi089.stderr:1800000 of 50000000 tuples (3%) done (elapsed 2.48 s, remaining 66.47 s) 2023-08-25T02:26:43.375 INFO:teuthology.orchestra.run.smithi089.stderr:1900000 of 50000000 tuples (3%) done (elapsed 2.67 s, remaining 67.67 s) 2023-08-25T02:26:43.471 INFO:teuthology.orchestra.run.smithi089.stderr:2000000 of 50000000 tuples (4%) done (elapsed 2.77 s, remaining 66.45 s) 2023-08-25T02:26:43.646 INFO:teuthology.orchestra.run.smithi089.stderr:2100000 of 50000000 tuples (4%) done (elapsed 2.94 s, remaining 67.14 s) 2023-08-25T02:26:43.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:43 smithi121 ceph-mon[100099]: pgmap v9: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 201 op/s 2023-08-25T02:26:43.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:43 smithi122 ceph-mon[100457]: pgmap v9: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 201 op/s 2023-08-25T02:26:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:43 smithi089 ceph-mon[98732]: pgmap v9: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 13 MiB/s wr, 201 op/s 2023-08-25T02:26:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2200000 of 50000000 tuples (4%) done (elapsed 3.13 s, remaining 68.00 s) 2023-08-25T02:26:43.936 INFO:teuthology.orchestra.run.smithi089.stderr:2300000 of 50000000 tuples (4%) done (elapsed 3.23 s, remaining 67.06 s) 2023-08-25T02:26:44.107 INFO:teuthology.orchestra.run.smithi089.stderr:2400000 of 50000000 tuples (4%) done (elapsed 3.41 s, remaining 67.54 s) 2023-08-25T02:26:44.277 INFO:teuthology.orchestra.run.smithi089.stderr:2500000 of 50000000 tuples (5%) done (elapsed 3.57 s, remaining 67.92 s) 2023-08-25T02:26:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2600000 of 50000000 tuples (5%) done (elapsed 3.67 s, remaining 66.96 s) 2023-08-25T02:26:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2700000 of 50000000 tuples (5%) done (elapsed 3.83 s, remaining 67.04 s) 2023-08-25T02:26:44.709 INFO:teuthology.orchestra.run.smithi089.stderr:2800000 of 50000000 tuples (5%) done (elapsed 4.01 s, remaining 67.54 s) 2023-08-25T02:26:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2900000 of 50000000 tuples (5%) done (elapsed 4.11 s, remaining 66.77 s) 2023-08-25T02:26:44.960 INFO:teuthology.orchestra.run.smithi089.stderr:3000000 of 50000000 tuples (6%) done (elapsed 4.26 s, remaining 66.70 s) 2023-08-25T02:26:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:3100000 of 50000000 tuples (6%) done (elapsed 4.43 s, remaining 67.10 s) 2023-08-25T02:26:45.200 INFO:teuthology.orchestra.run.smithi089.stderr:3200000 of 50000000 tuples (6%) done (elapsed 4.50 s, remaining 65.79 s) 2023-08-25T02:26:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:3300000 of 50000000 tuples (6%) done (elapsed 4.69 s, remaining 66.42 s) 2023-08-25T02:26:45.560 INFO:teuthology.orchestra.run.smithi089.stderr:3400000 of 50000000 tuples (6%) done (elapsed 4.86 s, remaining 66.59 s) 2023-08-25T02:26:45.631 INFO:teuthology.orchestra.run.smithi089.stderr:3500000 of 50000000 tuples (7%) done (elapsed 4.93 s, remaining 65.48 s) 2023-08-25T02:26:45.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:45 smithi121 ceph-mon[100099]: pgmap v10: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 12 MiB/s wr, 184 op/s 2023-08-25T02:26:45.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:45 smithi122 ceph-mon[100457]: pgmap v10: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 12 MiB/s wr, 184 op/s 2023-08-25T02:26:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:3600000 of 50000000 tuples (7%) done (elapsed 5.11 s, remaining 65.83 s) 2023-08-25T02:26:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:45 smithi089 ceph-mon[98732]: pgmap v10: 97 pgs: 97 active+clean; 91 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 12 MiB/s wr, 184 op/s 2023-08-25T02:26:45.878 INFO:teuthology.orchestra.run.smithi089.stderr:3700000 of 50000000 tuples (7%) done (elapsed 5.18 s, remaining 64.77 s) 2023-08-25T02:26:46.061 INFO:teuthology.orchestra.run.smithi089.stderr:3800000 of 50000000 tuples (7%) done (elapsed 5.36 s, remaining 65.15 s) 2023-08-25T02:26:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2753565927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 msgr2=0x7fe6b810c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2753565927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b810c650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000b0d0 tx=0x7fe6b000b3e0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2753565927 shutdown_connections 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2753565927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 0x7fe6b8105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2753565927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b810c650 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2753565927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 0x7fe6b80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2753565927 >> 172.21.15.89:0/2753565927 conn(0x7fe6b8075250 msgr2=0x7fe6b8075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2753565927 shutdown_connections 2023-08-25T02:26:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.063+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2753565927 wait complete. 2023-08-25T02:26:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 Processor -- start 2023-08-25T02:26:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 -- start start 2023-08-25T02:26:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 0x7fe6b819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 0x7fe6b81957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8195e40 con 0x7fe6b8100440 2023-08-25T02:26:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8195fb0 con 0x7fe6b8105840 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6be2ef700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8196170 con 0x7fe6b80ffa80 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6b77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6b77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48376/0 (socket says 172.21.15.89:48376) 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2029135178 learned_addr learned my addr 172.21.15.89:0/2029135178 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.064+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 0x7fe6b819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 0x7fe6b81957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2029135178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 msgr2=0x7fe6b819b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 0x7fe6b819b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2029135178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 msgr2=0x7fe6b81957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 0x7fe6b81957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2029135178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0009d00 con 0x7fe6b8100440 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00d980 tx=0x7fe6ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b57fa700 1 -- 172.21.15.89:0/2029135178 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac004220 con 0x7fe6b8100440 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b8196450 con 0x7fe6b8100440 2023-08-25T02:26:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b57fa700 1 -- 172.21.15.89:0/2029135178 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6ac007640 con 0x7fe6b8100440 2023-08-25T02:26:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6b57fa700 1 -- 172.21.15.89:0/2029135178 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac005610 con 0x7fe6b8100440 2023-08-25T02:26:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.065+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b81a2960 con 0x7fe6b8100440 2023-08-25T02:26:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.067+0000 7fe6b57fa700 1 -- 172.21.15.89:0/2029135178 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 21) v1 ==== 134847+0+0 (secure 0 0 0) 0x7fe6ac004750 con 0x7fe6b8100440 2023-08-25T02:26:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.067+0000 7fe6b57fa700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 0x7fe6a00a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.068+0000 7fe6b57fa700 1 -- 172.21.15.89:0/2029135178 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe6ac0cb6c0 con 0x7fe6b8100440 2023-08-25T02:26:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.068+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 0x7fe6a00a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.068+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 0x7fe6a00a2690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009be0 tx=0x7fe6a8009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 Processor -- start 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 -- start start 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 0x55b162d13cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d14650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162c75750 con 0x55b162d0f850 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162c66720 con 0x55b162d10170 2023-08-25T02:26:46.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.070+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162c64040 con 0x55b162d141f0 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48382/0 (socket says 172.21.15.89:48382) 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b7fff700 1 -- 172.21.15.89:0/2919640103 learned_addr learned my addr 172.21.15.89:0/2919640103 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 0x55b162d13cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d14650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2919640103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 msgr2=0x55b162d14650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d14650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2919640103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 msgr2=0x55b162d0fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 -- 172.21.15.89:0/2919640103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b162adf5d0 con 0x55b162d10170 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d14650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 0x55b162d13cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0c8cf0 tx=0x7fe6ac098630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=998406ce6fa80c34 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe69e7fc700 1 -- 172.21.15.89:0/2919640103 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac098c00 con 0x55b162d10170 2023-08-25T02:26:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe69e7fc700 1 -- 172.21.15.89:0/2919640103 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6ac09ca30 con 0x55b162d10170 2023-08-25T02:26:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.071+0000 7fe69e7fc700 1 -- 172.21.15.89:0/2919640103 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac0980f0 con 0x55b162d10170 2023-08-25T02:26:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 -- 172.21.15.89:0/2919640103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 msgr2=0x55b162d13cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 0x55b162d13cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0c8cf0 tx=0x7fe6ac098630 comp rx=0 tx=0).stop 2023-08-25T02:26:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 -- 172.21.15.89:0/2919640103 shutdown_connections 2023-08-25T02:26:46.084 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d14650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.085 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d0fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.085 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/2919640103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d10170 0x55b162d13cb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.085 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 -- 172.21.15.89:0/2919640103 >> 172.21.15.89:0/2919640103 conn(0x55b162c82080 msgr2=0x55b162d07d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.085 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 -- 172.21.15.89:0/2919640103 shutdown_connections 2023-08-25T02:26:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.072+0000 7fe6ca493b80 1 -- 172.21.15.89:0/2919640103 wait complete. 2023-08-25T02:26:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 -- start start 2023-08-25T02:26:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 Processor -- start 2023-08-25T02:26:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d1b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d19270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162c5f930 con 0x55b162d10170 2023-08-25T02:26:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162bfdb50 con 0x55b162d141f0 2023-08-25T02:26:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.073+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162bfd850 con 0x55b162d0f850 2023-08-25T02:26:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6ca493b80 10 client.0 _ll_register_callbacks cb 0x55b162b47a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48390/0 (socket says 172.21.15.89:48390) 2023-08-25T02:26:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 -- 172.21.15.89:0/3122525048 learned_addr learned my addr 172.21.15.89:0/3122525048 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d1b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d19270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 -- 172.21.15.89:0/3122525048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 msgr2=0x55b162d1b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d1b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 -- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 msgr2=0x55b162d19270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d19270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 -- 172.21.15.89:0/3122525048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b810cee0 con 0x55b162d10170 2023-08-25T02:26:46.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d1b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac095100 tx=0x7fe6ac0b81a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434d8d1fa3a3353d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6977fe700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac0a0a10 con 0x55b162d10170 2023-08-25T02:26:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe6ac098c00 con 0x55b162d10170 2023-08-25T02:26:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b162cdd160 con 0x55b162d10170 2023-08-25T02:26:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6ac0980f0 con 0x55b162d10170 2023-08-25T02:26:46.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6ca493b80 10 client.15333 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.074+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fe6b81a2960 con 0x55b162d10170 2023-08-25T02:26:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6ac07b080 con 0x55b162d10170 2023-08-25T02:26:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6ac0c9030 con 0x55b162d10170 2023-08-25T02:26:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 10 client.15333 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:46.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 10 client.15333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:26:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 20 client.15333 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d28fb0 0x55b162d2b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 4 client.15333 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:26:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b162c05b40 con 0x55b162d28fb0 2023-08-25T02:26:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d28fb0 0x55b162d2b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6977fe700 10 client.15333 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.075+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d28fb0 0x55b162d2b3a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e180641de26ebc8 server_cookie=7a683b2938691c4d in_seq=0 out_seq=0 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6977fe700 1 -- 172.21.15.89:0/3122525048 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b162c05b40 con 0x55b162d28fb0 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6977fe700 10 client.15333 handle_command_reply: tid=0 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 client.15333 shutdown 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 msgr2=0x55b162d18bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac095100 tx=0x7fe6ac0b81a0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 shutdown_connections 2023-08-25T02:26:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d28fb0 0x55b162d2b3a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d19270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d1b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3122525048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d10170 0x55b162d18bf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6977fe700 0 client.15333 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 >> 172.21.15.89:0/3122525048 conn(0x55b162c82080 msgr2=0x55b162d02820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 shutdown_connections 2023-08-25T02:26:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.076+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3122525048 wait complete. 2023-08-25T02:26:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.077+0000 7fe6ca493b80 20 client.15333 trim_cache size 0 max 16384 2023-08-25T02:26:46.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 Processor -- start 2023-08-25T02:26:46.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 -- start start 2023-08-25T02:26:46.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d20520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d07c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162ae18c0 con 0x55b162d11aa0 2023-08-25T02:26:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162d20a60 con 0x55b162d141f0 2023-08-25T02:26:46.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162d12ce0 con 0x55b162d0f850 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6bc88c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6bc88c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48398/0 (socket says 172.21.15.89:48398) 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6bc88c700 1 -- 172.21.15.89:0/3579563918 learned_addr learned my addr 172.21.15.89:0/3579563918 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d07c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.091+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d20520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 -- 172.21.15.89:0/3579563918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 msgr2=0x55b162d20520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d20520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 -- 172.21.15.89:0/3579563918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 msgr2=0x55b162d07c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d07c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 -- 172.21.15.89:0/3579563918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b162a8bd80 con 0x55b162d11aa0 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe6bc88c700 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009560 tx=0x7fe6b0009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92858bc5b122d96d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe69effd700 1 -- 172.21.15.89:0/3579563918 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000e040 con 0x55b162d11aa0 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe69effd700 1 -- 172.21.15.89:0/3579563918 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6b000bdd0 con 0x55b162d11aa0 2023-08-25T02:26:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.092+0000 7fe69effd700 1 -- 172.21.15.89:0/3579563918 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0005440 con 0x55b162d11aa0 2023-08-25T02:26:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3579563918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 msgr2=0x55b162d16590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009560 tx=0x7fe6b0009590 comp rx=0 tx=0).stop 2023-08-25T02:26:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3579563918 shutdown_connections 2023-08-25T02:26:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d11aa0 0x55b162d16590 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d0f850 0x55b162d20520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/3579563918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d141f0 0x55b162d07c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3579563918 >> 172.21.15.89:0/3579563918 conn(0x55b162c82080 msgr2=0x55b162d14d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3579563918 shutdown_connections 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- 172.21.15.89:0/3579563918 wait complete. 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.093+0000 7fe6ca493b80 1 -- start start 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 Processor -- start 2023-08-25T02:26:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 0x55b162d18ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162d17ca0 con 0x55b162d0f850 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162d17e10 con 0x55b162d11aa0 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b162d17fd0 con 0x55b162d141f0 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6ca493b80 10 client.0 _ll_register_callbacks cb 0x7fe6a80119e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48400/0 (socket says 172.21.15.89:48400) 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 -- 172.21.15.89:0/572190484 learned_addr learned my addr 172.21.15.89:0/572190484 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 0x55b162d18ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 -- 172.21.15.89:0/572190484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 msgr2=0x55b162d175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 -- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 msgr2=0x55b162d18ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 0x55b162d18ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b7fff700 1 -- 172.21.15.89:0/572190484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b0009320 con 0x55b162d0f850 2023-08-25T02:26:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.094+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 0x55b162d18ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:26:46.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6b7fff700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8005950 tx=0x7fe6a80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe67cba427751ef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:26:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8024040 con 0x55b162d0f850 2023-08-25T02:26:46.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe6a8029310 con 0x55b162d0f850 2023-08-25T02:26:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b162d00600 con 0x55b162d0f850 2023-08-25T02:26:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a80294d0 con 0x55b162d0f850 2023-08-25T02:26:46.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 10 client.15339 fetch_fsmap learned FSMap version 22 2023-08-25T02:26:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55b162a74c00 con 0x55b162d0f850 2023-08-25T02:26:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b162d00600 con 0x55b162d0f850 2023-08-25T02:26:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6a802f020 con 0x55b162d0f850 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 10 client.15339 fetch_fsmap finished waiting for FSMap version 22 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 10 client.15339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 20 client.15339 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d7e030 0x55b162d80450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 4 client.15339 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.095+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b162c58910 con 0x55b162d7e030 2023-08-25T02:26:46.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.096+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d7e030 0x55b162d80450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:46.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.096+0000 7fe6977fe700 10 client.15339 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:46.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.096+0000 7fe6b77fe700 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d7e030 0x55b162d80450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e64a5ab01dea053b server_cookie=b30d8af8b00a1a2f in_seq=0 out_seq=0 2023-08-25T02:26:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6977fe700 1 -- 172.21.15.89:0/572190484 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b162c58910 con 0x55b162d7e030 2023-08-25T02:26:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6977fe700 10 client.15339 handle_command_reply: tid=0 2023-08-25T02:26:46.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 client.15339 shutdown 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 msgr2=0x55b162d18760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8005950 tx=0x7fe6a80047c0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 shutdown_connections 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55b162d7e030 0x55b162d80450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b162d141f0 0x55b162d175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b162d0f850 0x55b162d18760 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6977fe700 0 client.15339 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 --2- 172.21.15.89:0/572190484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b162d11aa0 0x55b162d18ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 >> 172.21.15.89:0/572190484 conn(0x55b162c82080 msgr2=0x55b162d24c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 shutdown_connections 2023-08-25T02:26:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 1 -- 172.21.15.89:0/572190484 wait complete. 2023-08-25T02:26:46.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.265+0000 7fe6ca493b80 20 client.15339 trim_cache size 0 max 16384 2023-08-25T02:26:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:26:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:26:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:26:46.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:26:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 msgr2=0x7fe6a00a2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 0x7fe6a00a2690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009be0 tx=0x7fe6a8009260 comp rx=0 tx=0).stop 2023-08-25T02:26:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 msgr2=0x7fe6b819beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:46.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00d980 tx=0x7fe6ac00dc90 comp rx=0 tx=0).stop 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 shutdown_connections 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6a00a01d0 0x7fe6a00a2690 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6b8105840 0x7fe6b81957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6b80ffa80 0x7fe6b819b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 --2- 172.21.15.89:0/2029135178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6b8100440 0x7fe6b819beb0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 >> 172.21.15.89:0/2029135178 conn(0x7fe6b8075250 msgr2=0x7fe6b80fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:46.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 shutdown_connections 2023-08-25T02:26:46.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:46.266+0000 7fe6be2ef700 1 -- 172.21.15.89:0/2029135178 wait complete. 2023-08-25T02:26:46.352 INFO:teuthology.orchestra.run.smithi089.stderr:3900000 of 50000000 tuples (7%) done (elapsed 5.65 s, remaining 66.79 s) 2023-08-25T02:26:46.454 INFO:teuthology.orchestra.run.smithi089.stderr:4000000 of 50000000 tuples (8%) done (elapsed 5.75 s, remaining 66.13 s) 2023-08-25T02:26:46.601 INFO:teuthology.orchestra.run.smithi089.stderr:4100000 of 50000000 tuples (8%) done (elapsed 5.90 s, remaining 66.04 s) 2023-08-25T02:26:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:4200000 of 50000000 tuples (8%) done (elapsed 6.06 s, remaining 66.11 s) 2023-08-25T02:26:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:4300000 of 50000000 tuples (8%) done (elapsed 6.15 s, remaining 65.31 s) 2023-08-25T02:26:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:4400000 of 50000000 tuples (8%) done (elapsed 6.33 s, remaining 65.59 s) 2023-08-25T02:26:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:4500000 of 50000000 tuples (9%) done (elapsed 6.49 s, remaining 65.62 s) 2023-08-25T02:26:47.271 INFO:teuthology.orchestra.run.smithi089.stderr:4600000 of 50000000 tuples (9%) done (elapsed 6.57 s, remaining 64.83 s) 2023-08-25T02:26:47.447 INFO:teuthology.orchestra.run.smithi089.stderr:4700000 of 50000000 tuples (9%) done (elapsed 6.74 s, remaining 65.00 s) 2023-08-25T02:26:47.510 INFO:teuthology.orchestra.run.smithi089.stderr:4800000 of 50000000 tuples (9%) done (elapsed 6.81 s, remaining 64.11 s) 2023-08-25T02:26:47.696 INFO:teuthology.orchestra.run.smithi089.stderr:4900000 of 50000000 tuples (9%) done (elapsed 6.99 s, remaining 64.37 s) 2023-08-25T02:26:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:47 smithi121 ceph-mon[100099]: pgmap v11: 97 pgs: 97 active+clean; 187 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 28 MiB/s wr, 250 op/s 2023-08-25T02:26:47.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:47 smithi122 ceph-mon[100457]: pgmap v11: 97 pgs: 97 active+clean; 187 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 28 MiB/s wr, 250 op/s 2023-08-25T02:26:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:47 smithi089 ceph-mon[98732]: pgmap v11: 97 pgs: 97 active+clean; 187 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 28 MiB/s wr, 250 op/s 2023-08-25T02:26:47.880 INFO:teuthology.orchestra.run.smithi089.stderr:5000000 of 50000000 tuples (10%) done (elapsed 7.18 s, remaining 64.60 s) 2023-08-25T02:26:47.943 INFO:teuthology.orchestra.run.smithi089.stderr:5100000 of 50000000 tuples (10%) done (elapsed 7.24 s, remaining 63.74 s) 2023-08-25T02:26:48.128 INFO:teuthology.orchestra.run.smithi089.stderr:5200000 of 50000000 tuples (10%) done (elapsed 7.43 s, remaining 63.97 s) 2023-08-25T02:26:48.290 INFO:teuthology.orchestra.run.smithi089.stderr:5300000 of 50000000 tuples (10%) done (elapsed 7.59 s, remaining 64.00 s) 2023-08-25T02:26:48.376 INFO:teuthology.orchestra.run.smithi089.stderr:5400000 of 50000000 tuples (10%) done (elapsed 7.67 s, remaining 63.38 s) 2023-08-25T02:26:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:5500000 of 50000000 tuples (11%) done (elapsed 7.85 s, remaining 63.50 s) 2023-08-25T02:26:48.714 INFO:teuthology.orchestra.run.smithi089.stderr:5600000 of 50000000 tuples (11%) done (elapsed 8.01 s, remaining 63.52 s) 2023-08-25T02:26:48.803 INFO:teuthology.orchestra.run.smithi089.stderr:5700000 of 50000000 tuples (11%) done (elapsed 8.10 s, remaining 62.96 s) 2023-08-25T02:26:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:5800000 of 50000000 tuples (11%) done (elapsed 8.27 s, remaining 63.05 s) 2023-08-25T02:26:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:49 smithi121 ceph-mon[100099]: pgmap v12: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 71 MiB/s wr, 408 op/s 2023-08-25T02:26:49.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:49 smithi122 ceph-mon[100457]: pgmap v12: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 71 MiB/s wr, 408 op/s 2023-08-25T02:26:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:5900000 of 50000000 tuples (11%) done (elapsed 9.35 s, remaining 69.89 s) 2023-08-25T02:26:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:49 smithi089 ceph-mon[98732]: pgmap v12: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 71 MiB/s wr, 408 op/s 2023-08-25T02:26:50.124 INFO:teuthology.orchestra.run.smithi089.stderr:6000000 of 50000000 tuples (12%) done (elapsed 9.42 s, remaining 69.10 s) 2023-08-25T02:26:50.317 INFO:teuthology.orchestra.run.smithi089.stderr:6100000 of 50000000 tuples (12%) done (elapsed 9.62 s, remaining 69.20 s) 2023-08-25T02:26:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:6200000 of 50000000 tuples (12%) done (elapsed 9.68 s, remaining 68.41 s) 2023-08-25T02:26:50.556 INFO:teuthology.orchestra.run.smithi089.stderr:6300000 of 50000000 tuples (12%) done (elapsed 9.85 s, remaining 68.35 s) 2023-08-25T02:26:50.768 INFO:teuthology.orchestra.run.smithi089.stderr:6400000 of 50000000 tuples (12%) done (elapsed 10.07 s, remaining 68.58 s) 2023-08-25T02:26:50.841 INFO:teuthology.orchestra.run.smithi089.stderr:6500000 of 50000000 tuples (13%) done (elapsed 10.14 s, remaining 67.85 s) 2023-08-25T02:26:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:6600000 of 50000000 tuples (13%) done (elapsed 10.31 s, remaining 67.79 s) 2023-08-25T02:26:51.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:50 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:51.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:50 smithi089 ceph-mon[98732]: pgmap v13: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 71 MiB/s wr, 367 op/s 2023-08-25T02:26:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:6700000 of 50000000 tuples (13%) done (elapsed 10.41 s, remaining 67.28 s) 2023-08-25T02:26:51.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:50 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:51.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:50 smithi121 ceph-mon[100099]: pgmap v13: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 71 MiB/s wr, 367 op/s 2023-08-25T02:26:51.227 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:50 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:51.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:50 smithi122 ceph-mon[100457]: pgmap v13: 97 pgs: 97 active+clean; 439 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 71 MiB/s wr, 367 op/s 2023-08-25T02:26:51.294 INFO:teuthology.orchestra.run.smithi089.stderr:6800000 of 50000000 tuples (13%) done (elapsed 10.59 s, remaining 67.29 s) 2023-08-25T02:26:51.484 INFO:teuthology.orchestra.run.smithi089.stderr:6900000 of 50000000 tuples (13%) done (elapsed 10.78 s, remaining 67.34 s) 2023-08-25T02:26:51.583 INFO:teuthology.orchestra.run.smithi089.stderr:7000000 of 50000000 tuples (14%) done (elapsed 10.88 s, remaining 66.84 s) 2023-08-25T02:26:51.741 INFO:teuthology.orchestra.run.smithi089.stderr:7100000 of 50000000 tuples (14%) done (elapsed 11.04 s, remaining 66.70 s) 2023-08-25T02:26:51.914 INFO:teuthology.orchestra.run.smithi089.stderr:7200000 of 50000000 tuples (14%) done (elapsed 11.21 s, remaining 66.65 s) 2023-08-25T02:26:52.076 INFO:teuthology.orchestra.run.smithi089.stderr:7300000 of 50000000 tuples (14%) done (elapsed 11.37 s, remaining 66.53 s) 2023-08-25T02:26:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:7400000 of 50000000 tuples (14%) done (elapsed 11.45 s, remaining 65.92 s) 2023-08-25T02:26:52.312 INFO:teuthology.orchestra.run.smithi089.stderr:7500000 of 50000000 tuples (15%) done (elapsed 11.61 s, remaining 65.79 s) 2023-08-25T02:26:52.394 INFO:teuthology.orchestra.run.smithi089.stderr:7600000 of 50000000 tuples (15%) done (elapsed 11.69 s, remaining 65.23 s) 2023-08-25T02:26:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:7700000 of 50000000 tuples (15%) done (elapsed 11.86 s, remaining 65.17 s) 2023-08-25T02:26:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:7800000 of 50000000 tuples (15%) done (elapsed 12.04 s, remaining 65.13 s) 2023-08-25T02:26:52.821 INFO:teuthology.orchestra.run.smithi089.stderr:7900000 of 50000000 tuples (15%) done (elapsed 12.12 s, remaining 64.58 s) 2023-08-25T02:26:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:8000000 of 50000000 tuples (16%) done (elapsed 12.30 s, remaining 64.59 s) 2023-08-25T02:26:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:8100000 of 50000000 tuples (16%) done (elapsed 12.48 s, remaining 64.58 s) 2023-08-25T02:26:53.248 INFO:teuthology.orchestra.run.smithi089.stderr:8200000 of 50000000 tuples (16%) done (elapsed 12.55 s, remaining 63.95 s) 2023-08-25T02:26:53.438 INFO:teuthology.orchestra.run.smithi089.stderr:8300000 of 50000000 tuples (16%) done (elapsed 12.74 s, remaining 63.99 s) 2023-08-25T02:26:53.610 INFO:teuthology.orchestra.run.smithi089.stderr:8400000 of 50000000 tuples (16%) done (elapsed 12.91 s, remaining 63.93 s) 2023-08-25T02:26:53.678 INFO:teuthology.orchestra.run.smithi089.stderr:8500000 of 50000000 tuples (17%) done (elapsed 12.98 s, remaining 63.35 s) 2023-08-25T02:26:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:53 smithi121 ceph-mon[100099]: pgmap v14: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 183 MiB/s wr, 417 op/s 2023-08-25T02:26:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:53 smithi122 ceph-mon[100457]: pgmap v14: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 183 MiB/s wr, 417 op/s 2023-08-25T02:26:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:53 smithi089 ceph-mon[98732]: pgmap v14: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 183 MiB/s wr, 417 op/s 2023-08-25T02:26:53.872 INFO:teuthology.orchestra.run.smithi089.stderr:8600000 of 50000000 tuples (17%) done (elapsed 13.17 s, remaining 63.40 s) 2023-08-25T02:26:53.935 INFO:teuthology.orchestra.run.smithi089.stderr:8700000 of 50000000 tuples (17%) done (elapsed 13.23 s, remaining 62.82 s) 2023-08-25T02:26:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:8800000 of 50000000 tuples (17%) done (elapsed 13.41 s, remaining 62.78 s) 2023-08-25T02:26:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:8900000 of 50000000 tuples (17%) done (elapsed 13.60 s, remaining 62.81 s) 2023-08-25T02:26:54.377 INFO:teuthology.orchestra.run.smithi089.stderr:9000000 of 50000000 tuples (18%) done (elapsed 13.67 s, remaining 62.30 s) 2023-08-25T02:26:54.557 INFO:teuthology.orchestra.run.smithi089.stderr:9100000 of 50000000 tuples (18%) done (elapsed 13.86 s, remaining 62.27 s) 2023-08-25T02:26:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:9200000 of 50000000 tuples (18%) done (elapsed 13.95 s, remaining 61.88 s) 2023-08-25T02:26:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:9300000 of 50000000 tuples (18%) done (elapsed 14.12 s, remaining 61.79 s) 2023-08-25T02:26:54.995 INFO:teuthology.orchestra.run.smithi089.stderr:9400000 of 50000000 tuples (18%) done (elapsed 14.29 s, remaining 61.73 s) 2023-08-25T02:26:55.101 INFO:teuthology.orchestra.run.smithi089.stderr:9500000 of 50000000 tuples (19%) done (elapsed 14.40 s, remaining 61.38 s) 2023-08-25T02:26:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:9600000 of 50000000 tuples (19%) done (elapsed 14.55 s, remaining 61.23 s) 2023-08-25T02:26:55.429 INFO:teuthology.orchestra.run.smithi089.stderr:9700000 of 50000000 tuples (19%) done (elapsed 14.73 s, remaining 61.19 s) 2023-08-25T02:26:55.538 INFO:teuthology.orchestra.run.smithi089.stderr:9800000 of 50000000 tuples (19%) done (elapsed 14.84 s, remaining 60.86 s) 2023-08-25T02:26:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:55 smithi121 ceph-mon[100099]: pgmap v15: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 171 MiB/s wr, 274 op/s 2023-08-25T02:26:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:55 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:55 smithi122 ceph-mon[100457]: pgmap v15: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 171 MiB/s wr, 274 op/s 2023-08-25T02:26:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:55 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:55 smithi089 ceph-mon[98732]: pgmap v15: 97 pgs: 97 active+clean; 1.4 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 171 MiB/s wr, 274 op/s 2023-08-25T02:26:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:55 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:9900000 of 50000000 tuples (19%) done (elapsed 15.60 s, remaining 63.17 s) 2023-08-25T02:26:56.417 INFO:teuthology.orchestra.run.smithi089.stderr:10000000 of 50000000 tuples (20%) done (elapsed 15.71 s, remaining 62.86 s) 2023-08-25T02:26:56.984 INFO:teuthology.orchestra.run.smithi089.stderr:10100000 of 50000000 tuples (20%) done (elapsed 16.28 s, remaining 64.32 s) 2023-08-25T02:26:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:10200000 of 50000000 tuples (20%) done (elapsed 16.48 s, remaining 64.32 s) 2023-08-25T02:26:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:10300000 of 50000000 tuples (20%) done (elapsed 16.60 s, remaining 64.00 s) 2023-08-25T02:26:57.400 INFO:teuthology.orchestra.run.smithi089.stderr:10400000 of 50000000 tuples (20%) done (elapsed 16.70 s, remaining 63.58 s) 2023-08-25T02:26:57.504 INFO:teuthology.orchestra.run.smithi089.stderr:10500000 of 50000000 tuples (21%) done (elapsed 16.80 s, remaining 63.21 s) 2023-08-25T02:26:57.599 INFO:teuthology.orchestra.run.smithi089.stderr:10600000 of 50000000 tuples (21%) done (elapsed 16.90 s, remaining 62.80 s) 2023-08-25T02:26:57.717 INFO:teuthology.orchestra.run.smithi089.stderr:10700000 of 50000000 tuples (21%) done (elapsed 17.01 s, remaining 62.49 s) 2023-08-25T02:26:57.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:57 smithi121 ceph-mon[100099]: pgmap v16: 97 pgs: 97 active+clean; 1.8 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 210 MiB/s wr, 287 op/s 2023-08-25T02:26:57.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:57 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:57.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:57 smithi122 ceph-mon[100457]: pgmap v16: 97 pgs: 97 active+clean; 1.8 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 210 MiB/s wr, 287 op/s 2023-08-25T02:26:57.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:57 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:10800000 of 50000000 tuples (21%) done (elapsed 17.12 s, remaining 62.13 s) 2023-08-25T02:26:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:57 smithi089 ceph-mon[98732]: pgmap v16: 97 pgs: 97 active+clean; 1.8 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 210 MiB/s wr, 287 op/s 2023-08-25T02:26:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:57 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:10900000 of 50000000 tuples (21%) done (elapsed 17.22 s, remaining 61.77 s) 2023-08-25T02:26:57.987 INFO:teuthology.orchestra.run.smithi089.stderr:11000000 of 50000000 tuples (22%) done (elapsed 17.28 s, remaining 61.28 s) 2023-08-25T02:26:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:11100000 of 50000000 tuples (22%) done (elapsed 17.39 s, remaining 60.95 s) 2023-08-25T02:26:58.187 INFO:teuthology.orchestra.run.smithi089.stderr:11200000 of 50000000 tuples (22%) done (elapsed 17.48 s, remaining 60.57 s) 2023-08-25T02:26:58.295 INFO:teuthology.orchestra.run.smithi089.stderr:11300000 of 50000000 tuples (22%) done (elapsed 17.59 s, remaining 60.25 s) 2023-08-25T02:26:58.428 INFO:teuthology.orchestra.run.smithi089.stderr:11400000 of 50000000 tuples (22%) done (elapsed 17.73 s, remaining 60.02 s) 2023-08-25T02:26:58.911 INFO:teuthology.orchestra.run.smithi089.stderr:11500000 of 50000000 tuples (23%) done (elapsed 18.21 s, remaining 60.96 s) 2023-08-25T02:26:59.168 INFO:teuthology.orchestra.run.smithi089.stderr:11600000 of 50000000 tuples (23%) done (elapsed 18.47 s, remaining 61.13 s) 2023-08-25T02:26:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:11700000 of 50000000 tuples (23%) done (elapsed 18.57 s, remaining 60.79 s) 2023-08-25T02:26:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 -- 172.21.15.89:0/122355504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 msgr2=0x7f3644106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 --2- 172.21.15.89:0/122355504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f3644106cb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f36400099c0 tx=0x7f3640009cd0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 -- 172.21.15.89:0/122355504 shutdown_connections 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 --2- 172.21.15.89:0/122355504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36440686d0 0x7f364410d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 --2- 172.21.15.89:0/122355504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f3644106cb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 --2- 172.21.15.89:0/122355504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3644067d10 0x7f3644068190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 -- 172.21.15.89:0/122355504 >> 172.21.15.89:0/122355504 conn(0x7f3644075250 msgr2=0x7f3644075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 -- 172.21.15.89:0/122355504 shutdown_connections 2023-08-25T02:26:59.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.381+0000 7f364b44a700 1 -- 172.21.15.89:0/122355504 wait complete. 2023-08-25T02:26:59.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 Processor -- start 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 -- start start 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36440686d0 0x7f364419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644196e20 con 0x7f36441068d0 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644196f90 con 0x7f36440686d0 2023-08-25T02:26:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f364b44a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644197150 con 0x7f3644067d10 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f36499e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.382+0000 7f36491e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36491e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40348/0 (socket says 172.21.15.89:40348) 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48416/0 (socket says 172.21.15.89:48416) 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36491e6700 1 -- 172.21.15.89:0/3100496455 learned_addr learned my addr 172.21.15.89:0/3100496455 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36489e5700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36440686d0 0x7f364419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 -- 172.21.15.89:0/3100496455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 msgr2=0x7f364419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 -- 172.21.15.89:0/3100496455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36440686d0 msgr2=0x7f364419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36440686d0 0x7f364419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 -- 172.21.15.89:0/3100496455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400095f0 con 0x7f36441068d0 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36491e6700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f36499e7700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f363c007f10 tx=0x7f363c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f363a7fc700 1 -- 172.21.15.89:0/3100496455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c004db0 con 0x7f36441068d0 2023-08-25T02:26:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3644197430 con 0x7f36441068d0 2023-08-25T02:26:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f363a7fc700 1 -- 172.21.15.89:0/3100496455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f363c00f040 con 0x7f36441068d0 2023-08-25T02:26:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f363a7fc700 1 -- 172.21.15.89:0/3100496455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c005990 con 0x7f36441068d0 2023-08-25T02:26:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.383+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36441a3940 con 0x7f36441068d0 2023-08-25T02:26:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.385+0000 7f363a7fc700 1 -- 172.21.15.89:0/3100496455 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 21) v1 ==== 134847+0+0 (secure 0 0 0) 0x7f363c00db20 con 0x7f36441068d0 2023-08-25T02:26:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.386+0000 7f363a7fc700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 0x7f36300a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.386+0000 7f363a7fc700 1 -- 172.21.15.89:0/3100496455 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f363c0cb080 con 0x7f36441068d0 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.386+0000 7f36491e6700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 0x7f36300a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.387+0000 7f36491e6700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 0x7f36300a2610 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3640009f50 tx=0x7f3640005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 Processor -- start 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 -- start start 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41daf40 0x55bcb41db320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41d5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb413d750 con 0x55bcb41db860 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb412e720 con 0x55bcb41daf40 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36575eeb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb412c040 con 0x55bcb41d5990 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36489e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36499e7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41d5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36489e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48422/0 (socket says 172.21.15.89:48422) 2023-08-25T02:26:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36489e5700 1 -- 172.21.15.89:0/1500404563 learned_addr learned my addr 172.21.15.89:0/1500404563 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.388+0000 7f36491e6700 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41daf40 0x55bcb41db320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 -- 172.21.15.89:0/1500404563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41daf40 msgr2=0x55bcb41db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41daf40 0x55bcb41db320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 -- 172.21.15.89:0/1500404563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 msgr2=0x55bcb41d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 -- 172.21.15.89:0/1500404563 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb3fa75d0 con 0x55bcb41d5990 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f36499e7700 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41d5dd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f363c0c86b0 tx=0x7f363c094900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1226790cef3e5cd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f362f7fe700 1 -- 172.21.15.89:0/1500404563 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c097f20 con 0x55bcb41d5990 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f362f7fe700 1 -- 172.21.15.89:0/1500404563 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f363c098cc0 con 0x55bcb41d5990 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.389+0000 7f362f7fe700 1 -- 172.21.15.89:0/1500404563 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c09c1b0 con 0x55bcb41d5990 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- 172.21.15.89:0/1500404563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 msgr2=0x55bcb41d5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41d5dd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f363c0c86b0 tx=0x7f363c094900 comp rx=0 tx=0).stop 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- 172.21.15.89:0/1500404563 shutdown_connections 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41daf40 0x55bcb41db320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41d5dd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1500404563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- 172.21.15.89:0/1500404563 >> 172.21.15.89:0/1500404563 conn(0x55bcb414a080 msgr2=0x55bcb41cfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- 172.21.15.89:0/1500404563 shutdown_connections 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- 172.21.15.89:0/1500404563 wait complete. 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 -- start start 2023-08-25T02:26:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.390+0000 7f36575eeb80 1 Processor -- start 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41de990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41db860 0x55bcb41df410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb4127930 con 0x55bcb41daf40 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb40c5b50 con 0x55bcb41db860 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb40c5850 con 0x55bcb41d5990 2023-08-25T02:26:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36575eeb80 10 client.0 _ll_register_callbacks cb 0x55bcb400fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36489e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36489e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48428/0 (socket says 172.21.15.89:48428) 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41de990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36489e5700 1 -- 172.21.15.89:0/3730828478 learned_addr learned my addr 172.21.15.89:0/3730828478 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36499e7700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41db860 0x55bcb41df410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 -- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41db860 msgr2=0x55bcb41df410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41db860 0x55bcb41df410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 -- 172.21.15.89:0/3730828478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 msgr2=0x55bcb41deed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36491e6700 1 -- 172.21.15.89:0/3730828478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36441a3940 con 0x55bcb41d5990 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.391+0000 7f36489e5700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36491e6700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41de990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f364001a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81a8889ab8c8dde server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:26:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002b050 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcb41a5160 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f3640003dd0 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3640003f90 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 10 client.24997 fetch_fsmap learned FSMap version 26 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55bcb3f53d80 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364003e020 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 26) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3640005230 con 0x55bcb41d5990 2023-08-25T02:26:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 10 client.24997 fetch_fsmap finished waiting for FSMap version 26 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 10 client.24997 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 20 client.24997 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb41ef2b0 0x55bcb41f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 4 client.24997 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.392+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcb40cdb40 con 0x55bcb41ef2b0 2023-08-25T02:26:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.393+0000 7f36489e5700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb41ef2b0 0x55bcb41f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.404 INFO:teuthology.orchestra.run.smithi089.stderr:11800000 of 50000000 tuples (23%) done (elapsed 18.69 s, remaining 60.51 s) 2023-08-25T02:26:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.393+0000 7f362cff9700 10 client.24997 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.393+0000 7f36489e5700 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb41ef2b0 0x55bcb41f16d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73c01585c08b89b6 server_cookie=f1a70d65d628db7f in_seq=0 out_seq=0 2023-08-25T02:26:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f362cff9700 1 -- 172.21.15.89:0/3730828478 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bcb40cdb40 con 0x55bcb41ef2b0 2023-08-25T02:26:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f362cff9700 10 client.24997 handle_command_reply: tid=0 2023-08-25T02:26:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 client.24997 shutdown 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 msgr2=0x55bcb41de990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41de990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f364001a050 comp rx=0 tx=0).stop 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 shutdown_connections 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb41ef2b0 0x55bcb41f16d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41db860 0x55bcb41df410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f362cff9700 0 client.24997 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41d5990 0x55bcb41de990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3730828478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41daf40 0x55bcb41deed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 >> 172.21.15.89:0/3730828478 conn(0x55bcb414a080 msgr2=0x55bcb41ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 shutdown_connections 2023-08-25T02:26:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 1 -- 172.21.15.89:0/3730828478 wait complete. 2023-08-25T02:26:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.394+0000 7f36575eeb80 20 client.24997 trim_cache size 0 max 16384 2023-08-25T02:26:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 Processor -- start 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 -- start start 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb3fa98c0 con 0x55bcb41db860 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb41db320 con 0x55bcb41d5990 2023-08-25T02:26:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.421+0000 7f36575eeb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb41ebba0 con 0x55bcb41daf40 2023-08-25T02:26:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36499e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36499e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48438/0 (socket says 172.21.15.89:48438) 2023-08-25T02:26:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36499e7700 1 -- 172.21.15.89:0/3950329679 learned_addr learned my addr 172.21.15.89:0/3950329679 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36491e6700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 -- 172.21.15.89:0/3950329679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 msgr2=0x55bcb41d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 -- 172.21.15.89:0/3950329679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 msgr2=0x55bcb41df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 -- 172.21.15.89:0/3950329679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb3f3cc00 con 0x55bcb41daf40 2023-08-25T02:26:59.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36499e7700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:26:59.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f36489e5700 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41e6240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55bcb41f1f80 tx=0x7f363400a380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8605d712bac0f7dd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f362ffff700 1 -- 172.21.15.89:0/3950329679 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634015040 con 0x55bcb41daf40 2023-08-25T02:26:59.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f362ffff700 1 -- 172.21.15.89:0/3950329679 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3634016040 con 0x55bcb41daf40 2023-08-25T02:26:59.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.422+0000 7f362ffff700 1 -- 172.21.15.89:0/3950329679 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363401d7c0 con 0x55bcb41daf40 2023-08-25T02:26:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- 172.21.15.89:0/3950329679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 msgr2=0x55bcb41e6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41e6240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55bcb41f1f80 tx=0x7f363400a380 comp rx=0 tx=0).stop 2023-08-25T02:26:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- 172.21.15.89:0/3950329679 shutdown_connections 2023-08-25T02:26:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41df240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 --2- 172.21.15.89:0/3950329679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41e6240 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- 172.21.15.89:0/3950329679 >> 172.21.15.89:0/3950329679 conn(0x55bcb414a080 msgr2=0x55bcb41d8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- 172.21.15.89:0/3950329679 shutdown_connections 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- 172.21.15.89:0/3950329679 wait complete. 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.423+0000 7f36575eeb80 1 -- start start 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 Processor -- start 2023-08-25T02:26:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb41d7bc0 con 0x55bcb41db860 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb41d7d30 con 0x55bcb41d5990 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb41d7ef0 con 0x55bcb41daf40 2023-08-25T02:26:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36575eeb80 10 client.0 _ll_register_callbacks cb 0x7f364000f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:26:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36489e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36489e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40378/0 (socket says 172.21.15.89:40378) 2023-08-25T02:26:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36489e5700 1 -- 172.21.15.89:0/1263063831 learned_addr learned my addr 172.21.15.89:0/1263063831 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:26:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36499e7700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.424+0000 7f36491e6700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 -- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 msgr2=0x55bcb41d6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 -- 172.21.15.89:0/1263063831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 msgr2=0x55bcb41d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 -- 172.21.15.89:0/1263063831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcb3fa75d0 con 0x55bcb41daf40 2023-08-25T02:26:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36491e6700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:26:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36489e5700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3634022040 tx=0x7f3634010df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5404eb3dcdce16b4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:26:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f362cff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:26:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634005460 con 0x55bcb41daf40 2023-08-25T02:26:59.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f36340055c0 con 0x55bcb41daf40 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcb41c8600 con 0x55bcb41daf40 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f363401f550 con 0x55bcb41daf40 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36575eeb80 10 client.25003 fetch_fsmap learned FSMap version 26 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f363401fa00 con 0x55bcb41daf40 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.425+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bcb41c8600 con 0x55bcb41daf40 2023-08-25T02:26:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 26) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f36340288a0 con 0x55bcb41daf40 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 10 client.25003 fetch_fsmap finished waiting for FSMap version 26 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 10 client.25003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 20 client.25003 populate_metadata read hostname 'smithi089' 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb4243f00 0x55bcb4246320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 4 client.25003 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bcb4120910 con 0x55bcb4243f00 2023-08-25T02:26:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36491e6700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb4243f00 0x55bcb4246320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:26:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f362cff9700 10 client.25003 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.426+0000 7f36491e6700 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb4243f00 0x55bcb4246320 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff22fbf5a70f8ef4 server_cookie=f20f27513983dc7d in_seq=0 out_seq=0 2023-08-25T02:26:59.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f362cff9700 1 -- 172.21.15.89:0/1263063831 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bcb4120910 con 0x55bcb4243f00 2023-08-25T02:26:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f362cff9700 10 client.25003 handle_command_reply: tid=0 2023-08-25T02:26:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 client.25003 shutdown 2023-08-25T02:26:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 msgr2=0x55bcb41d6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3634022040 tx=0x7f3634010df0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 shutdown_connections 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55bcb4243f00 0x55bcb4246320 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bcb41d5990 0x55bcb41d6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bcb41db860 0x55bcb41d74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 --2- 172.21.15.89:0/1263063831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcb41daf40 0x55bcb41d6dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 >> 172.21.15.89:0/1263063831 conn(0x55bcb414a080 msgr2=0x55bcb41dbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.443+0000 7f362cff9700 0 client.25003 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 shutdown_connections 2023-08-25T02:26:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f36575eeb80 1 -- 172.21.15.89:0/1263063831 wait complete. 2023-08-25T02:26:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f36575eeb80 20 client.25003 trim_cache size 0 max 16384 2023-08-25T02:26:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:26:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:26:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:26:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:26:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 msgr2=0x7f36300a2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 0x7f36300a2610 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3640009f50 tx=0x7f3640005fb0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 msgr2=0x7f36441967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f363c007f10 tx=0x7f363c00d3b0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 shutdown_connections 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f36300a0150 0x7f36300a2610 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441068d0 0x7f36441967b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644067d10 0x7f364419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 --2- 172.21.15.89:0/3100496455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36440686d0 0x7f364419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:26:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.444+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 >> 172.21.15.89:0/3100496455 conn(0x7f3644075250 msgr2=0x7f36440fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:26:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.445+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 shutdown_connections 2023-08-25T02:26:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:26:59.445+0000 7f364b44a700 1 -- 172.21.15.89:0/3100496455 wait complete. 2023-08-25T02:26:59.518 INFO:teuthology.orchestra.run.smithi089.stderr:11900000 of 50000000 tuples (23%) done (elapsed 18.82 s, remaining 60.24 s) 2023-08-25T02:26:59.606 INFO:teuthology.orchestra.run.smithi089.stderr:12000000 of 50000000 tuples (24%) done (elapsed 18.90 s, remaining 59.86 s) 2023-08-25T02:26:59.699 INFO:teuthology.orchestra.run.smithi089.stderr:12100000 of 50000000 tuples (24%) done (elapsed 19.00 s, remaining 59.50 s) 2023-08-25T02:26:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:59 smithi121 ceph-mon[100099]: pgmap v17: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 251 MiB/s wr, 257 op/s 2023-08-25T02:26:59.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:26:59 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:59.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:59 smithi122 ceph-mon[100457]: pgmap v17: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 251 MiB/s wr, 257 op/s 2023-08-25T02:26:59.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:26:59 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:59.819 INFO:teuthology.orchestra.run.smithi089.stderr:12200000 of 50000000 tuples (24%) done (elapsed 19.12 s, remaining 59.23 s) 2023-08-25T02:26:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:59 smithi089 ceph-mon[98732]: pgmap v17: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 251 MiB/s wr, 257 op/s 2023-08-25T02:26:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:26:59 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:26:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:12300000 of 50000000 tuples (24%) done (elapsed 19.22 s, remaining 58.90 s) 2023-08-25T02:27:00.027 INFO:teuthology.orchestra.run.smithi089.stderr:12400000 of 50000000 tuples (24%) done (elapsed 19.33 s, remaining 58.60 s) 2023-08-25T02:27:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:12500000 of 50000000 tuples (25%) done (elapsed 19.51 s, remaining 58.54 s) 2023-08-25T02:27:00.677 INFO:teuthology.orchestra.run.smithi089.stderr:12600000 of 50000000 tuples (25%) done (elapsed 19.97 s, remaining 59.29 s) 2023-08-25T02:27:01.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:01 smithi089 ceph-mon[98732]: pgmap v18: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 209 MiB/s wr, 99 op/s 2023-08-25T02:27:01.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:27:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:27:00.895+0000 7fd1bb0b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:27:01.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:27:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:27:00.896+0000 7fd1bb0b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:27:01.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:27:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:27:00.896+0000 7fd1bb0b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:27:01.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:27:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:27:00.896+0000 7fd1bb0b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:27:01.330 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:27:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:27:00.896+0000 7fd1bb0b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-08-25T02:27:01.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:01 smithi121 ceph-mon[100099]: pgmap v18: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 209 MiB/s wr, 99 op/s 2023-08-25T02:27:01.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:01 smithi122 ceph-mon[100457]: pgmap v18: 97 pgs: 97 active+clean; 2.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 209 MiB/s wr, 99 op/s 2023-08-25T02:27:01.482 INFO:teuthology.orchestra.run.smithi089.stderr:12700000 of 50000000 tuples (25%) done (elapsed 20.78 s, remaining 61.03 s) 2023-08-25T02:27:02.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:02 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:02 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:02.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:02 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:03.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:03 smithi121 ceph-mon[100099]: pgmap v19: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 233 MiB/s wr, 280 op/s 2023-08-25T02:27:03.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:03 smithi122 ceph-mon[100457]: pgmap v19: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 233 MiB/s wr, 280 op/s 2023-08-25T02:27:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:03 smithi089 ceph-mon[98732]: pgmap v19: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 233 MiB/s wr, 280 op/s 2023-08-25T02:27:04.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:04 smithi121 ceph-mon[100099]: mgrmap e22: x(active, since 32s), standbys: y, z 2023-08-25T02:27:04.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:04 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:04.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:04 smithi122 ceph-mon[100457]: mgrmap e22: x(active, since 32s), standbys: y, z 2023-08-25T02:27:04.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:04 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:04.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:04 smithi089 ceph-mon[98732]: mgrmap e22: x(active, since 32s), standbys: y, z 2023-08-25T02:27:04.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:04 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:27:05.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:05 smithi121 ceph-mon[100099]: pgmap v20: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 120 MiB/s wr, 230 op/s 2023-08-25T02:27:05.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:05 smithi122 ceph-mon[100457]: pgmap v20: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 120 MiB/s wr, 230 op/s 2023-08-25T02:27:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:05 smithi089 ceph-mon[98732]: pgmap v20: 97 pgs: 97 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 120 MiB/s wr, 230 op/s 2023-08-25T02:27:05.665 INFO:teuthology.orchestra.run.smithi089.stderr:12800000 of 50000000 tuples (25%) done (elapsed 24.96 s, remaining 72.55 s) 2023-08-25T02:27:06.433 INFO:teuthology.orchestra.run.smithi089.stderr:12900000 of 50000000 tuples (25%) done (elapsed 25.73 s, remaining 74.00 s) 2023-08-25T02:27:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:13000000 of 50000000 tuples (26%) done (elapsed 26.14 s, remaining 74.39 s) 2023-08-25T02:27:07.283 INFO:teuthology.orchestra.run.smithi089.stderr:13100000 of 50000000 tuples (26%) done (elapsed 26.58 s, remaining 74.87 s) 2023-08-25T02:27:07.395 INFO:teuthology.orchestra.run.smithi089.stderr:13200000 of 50000000 tuples (26%) done (elapsed 26.69 s, remaining 74.42 s) 2023-08-25T02:27:07.496 INFO:teuthology.orchestra.run.smithi089.stderr:13300000 of 50000000 tuples (26%) done (elapsed 26.79 s, remaining 73.94 s) 2023-08-25T02:27:07.609 INFO:teuthology.orchestra.run.smithi089.stderr:13400000 of 50000000 tuples (26%) done (elapsed 26.91 s, remaining 73.49 s) 2023-08-25T02:27:07.674 INFO:teuthology.orchestra.run.smithi089.stderr:13500000 of 50000000 tuples (27%) done (elapsed 26.97 s, remaining 72.92 s) 2023-08-25T02:27:07.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:07 smithi121 ceph-mon[100099]: pgmap v21: 97 pgs: 97 active+clean; 2.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 122 MiB/s wr, 543 op/s 2023-08-25T02:27:07.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:07 smithi122 ceph-mon[100457]: pgmap v21: 97 pgs: 97 active+clean; 2.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 122 MiB/s wr, 543 op/s 2023-08-25T02:27:07.789 INFO:teuthology.orchestra.run.smithi089.stderr:13600000 of 50000000 tuples (27%) done (elapsed 27.09 s, remaining 72.50 s) 2023-08-25T02:27:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:07 smithi089 ceph-mon[98732]: pgmap v21: 97 pgs: 97 active+clean; 2.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 122 MiB/s wr, 543 op/s 2023-08-25T02:27:07.891 INFO:teuthology.orchestra.run.smithi089.stderr:13700000 of 50000000 tuples (27%) done (elapsed 27.19 s, remaining 72.04 s) 2023-08-25T02:27:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:13800000 of 50000000 tuples (27%) done (elapsed 27.96 s, remaining 73.35 s) 2023-08-25T02:27:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:13900000 of 50000000 tuples (27%) done (elapsed 28.07 s, remaining 72.90 s) 2023-08-25T02:27:08.891 INFO:teuthology.orchestra.run.smithi089.stderr:14000000 of 50000000 tuples (28%) done (elapsed 28.19 s, remaining 72.48 s) 2023-08-25T02:27:08.969 INFO:teuthology.orchestra.run.smithi089.stderr:14100000 of 50000000 tuples (28%) done (elapsed 28.27 s, remaining 71.97 s) 2023-08-25T02:27:09.143 INFO:teuthology.orchestra.run.smithi089.stderr:14200000 of 50000000 tuples (28%) done (elapsed 28.44 s, remaining 71.70 s) 2023-08-25T02:27:09.253 INFO:teuthology.orchestra.run.smithi089.stderr:14300000 of 50000000 tuples (28%) done (elapsed 28.55 s, remaining 71.28 s) 2023-08-25T02:27:09.341 INFO:teuthology.orchestra.run.smithi089.stderr:14400000 of 50000000 tuples (28%) done (elapsed 28.64 s, remaining 70.80 s) 2023-08-25T02:27:09.446 INFO:teuthology.orchestra.run.smithi089.stderr:14500000 of 50000000 tuples (29%) done (elapsed 28.74 s, remaining 70.37 s) 2023-08-25T02:27:09.570 INFO:teuthology.orchestra.run.smithi089.stderr:14600000 of 50000000 tuples (29%) done (elapsed 28.87 s, remaining 69.99 s) 2023-08-25T02:27:09.676 INFO:teuthology.orchestra.run.smithi089.stderr:14700000 of 50000000 tuples (29%) done (elapsed 28.97 s, remaining 69.58 s) 2023-08-25T02:27:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:09 smithi121 ceph-mon[100099]: pgmap v22: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 MiB/s wr, 533 op/s 2023-08-25T02:27:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:09 smithi122 ceph-mon[100457]: pgmap v22: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 MiB/s wr, 533 op/s 2023-08-25T02:27:09.769 INFO:teuthology.orchestra.run.smithi089.stderr:14800000 of 50000000 tuples (29%) done (elapsed 29.07 s, remaining 69.13 s) 2023-08-25T02:27:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:09 smithi089 ceph-mon[98732]: pgmap v22: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 MiB/s wr, 533 op/s 2023-08-25T02:27:09.854 INFO:teuthology.orchestra.run.smithi089.stderr:14900000 of 50000000 tuples (29%) done (elapsed 29.15 s, remaining 68.67 s) 2023-08-25T02:27:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:15000000 of 50000000 tuples (30%) done (elapsed 29.30 s, remaining 68.37 s) 2023-08-25T02:27:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:15100000 of 50000000 tuples (30%) done (elapsed 29.37 s, remaining 67.87 s) 2023-08-25T02:27:10.183 INFO:teuthology.orchestra.run.smithi089.stderr:15200000 of 50000000 tuples (30%) done (elapsed 29.48 s, remaining 67.50 s) 2023-08-25T02:27:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:15300000 of 50000000 tuples (30%) done (elapsed 29.62 s, remaining 67.18 s) 2023-08-25T02:27:10.432 INFO:teuthology.orchestra.run.smithi089.stderr:15400000 of 50000000 tuples (30%) done (elapsed 29.73 s, remaining 66.80 s) 2023-08-25T02:27:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:15500000 of 50000000 tuples (31%) done (elapsed 29.81 s, remaining 66.35 s) 2023-08-25T02:27:10.614 INFO:teuthology.orchestra.run.smithi089.stderr:15600000 of 50000000 tuples (31%) done (elapsed 29.91 s, remaining 65.96 s) 2023-08-25T02:27:10.713 INFO:teuthology.orchestra.run.smithi089.stderr:15700000 of 50000000 tuples (31%) done (elapsed 30.01 s, remaining 65.56 s) 2023-08-25T02:27:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:15800000 of 50000000 tuples (31%) done (elapsed 30.18 s, remaining 65.33 s) 2023-08-25T02:27:11.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:11 smithi121 ceph-mon[100099]: pgmap v23: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 26 MiB/s wr, 497 op/s 2023-08-25T02:27:11.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:11 smithi122 ceph-mon[100457]: pgmap v23: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 26 MiB/s wr, 497 op/s 2023-08-25T02:27:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:15900000 of 50000000 tuples (31%) done (elapsed 30.82 s, remaining 66.09 s) 2023-08-25T02:27:11.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:11 smithi089 ceph-mon[98732]: pgmap v23: 97 pgs: 97 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 26 MiB/s wr, 497 op/s 2023-08-25T02:27:11.593 INFO:teuthology.orchestra.run.smithi089.stderr:16000000 of 50000000 tuples (32%) done (elapsed 30.89 s, remaining 65.64 s) 2023-08-25T02:27:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:16100000 of 50000000 tuples (32%) done (elapsed 31.04 s, remaining 65.37 s) 2023-08-25T02:27:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:16200000 of 50000000 tuples (32%) done (elapsed 31.60 s, remaining 65.93 s) 2023-08-25T02:27:12.445 INFO:teuthology.orchestra.run.smithi089.stderr:16300000 of 50000000 tuples (32%) done (elapsed 31.74 s, remaining 65.63 s) 2023-08-25T02:27:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- 172.21.15.89:0/3507624566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81068d0 msgr2=0x7f90f8106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 --2- 172.21.15.89:0/3507624566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81068d0 0x7f90f8106cb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009990 tx=0x7f90ec009ca0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- 172.21.15.89:0/3507624566 shutdown_connections 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 --2- 172.21.15.89:0/3507624566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 0x7f90f810d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 --2- 172.21.15.89:0/3507624566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81068d0 0x7f90f8106cb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 --2- 172.21.15.89:0/3507624566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81008c0 0x7f90f8100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- 172.21.15.89:0/3507624566 >> 172.21.15.89:0/3507624566 conn(0x7f90f80fc370 msgr2=0x7f90f80fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- 172.21.15.89:0/3507624566 shutdown_connections 2023-08-25T02:27:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- 172.21.15.89:0/3507624566 wait complete. 2023-08-25T02:27:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 Processor -- start 2023-08-25T02:27:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.565+0000 7f90ff05e700 1 -- start start 2023-08-25T02:27:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 0x7f90f8073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81068d0 0x7f90f81a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f81a5090 con 0x7f90f81008c0 2023-08-25T02:27:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f81a51d0 con 0x7f90f8101280 2023-08-25T02:27:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90ff05e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f81a5340 con 0x7f90f81068d0 2023-08-25T02:27:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48470/0 (socket says 172.21.15.89:48470) 2023-08-25T02:27:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 -- 172.21.15.89:0/662664739 learned_addr learned my addr 172.21.15.89:0/662664739 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81068d0 0x7f90f81a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90f7fff700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 0x7f90f8073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 -- 172.21.15.89:0/662664739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81068d0 msgr2=0x7f90f81a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81068d0 0x7f90f81a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 -- 172.21.15.89:0/662664739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 msgr2=0x7f90f8073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 0x7f90f8073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 -- 172.21.15.89:0/662664739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0095f0 con 0x7f90f81008c0 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0095c0 tx=0x7f90ec00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.566+0000 7f90f5ffb700 1 -- 172.21.15.89:0/662664739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec01d070 con 0x7f90f81008c0 2023-08-25T02:27:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.567+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f81a5620 con 0x7f90f81008c0 2023-08-25T02:27:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.567+0000 7f90f5ffb700 1 -- 172.21.15.89:0/662664739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90ec0054f0 con 0x7f90f81008c0 2023-08-25T02:27:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.567+0000 7f90f5ffb700 1 -- 172.21.15.89:0/662664739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec004a70 con 0x7f90f81008c0 2023-08-25T02:27:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.567+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f81a5b30 con 0x7f90f81008c0 2023-08-25T02:27:12.577 INFO:teuthology.orchestra.run.smithi089.stderr:16400000 of 50000000 tuples (32%) done (elapsed 31.87 s, remaining 65.29 s) 2023-08-25T02:27:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f90f5ffb700 1 -- 172.21.15.89:0/662664739 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 134904+0+0 (secure 0 0 0) 0x7f90ec004bd0 con 0x7f90f81008c0 2023-08-25T02:27:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f910b202b80 1 Processor -- start 2023-08-25T02:27:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f910b202b80 1 -- start start 2023-08-25T02:27:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f910b202b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f910b202b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746bdc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.570+0000 7f910b202b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bdcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f910b202b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746b44750 con 0x555746be1f40 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f910b202b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746b35720 con 0x555746bdc990 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f910b202b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746b33040 con 0x555746be2860 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90f5ffb700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 0x7f90e80a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48476/0 (socket says 172.21.15.89:48476) 2023-08-25T02:27:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 -- 172.21.15.89:0/2929058968 learned_addr learned my addr 172.21.15.89:0/2929058968 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90f5ffb700 1 -- 172.21.15.89:0/662664739 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f90ec0cc660 con 0x7f90f81008c0 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 0x7f90e80a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746bdc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90f7fff700 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bdcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 -- 172.21.15.89:0/2929058968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 msgr2=0x555746bdc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746bdc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 -- 172.21.15.89:0/2929058968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 msgr2=0x555746bdcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bdcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 -- 172.21.15.89:0/2929058968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557469ae5d0 con 0x555746be1f40 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f90f0007820 tx=0x7f90f0007b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fe2736b78c0296 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90e2ffd700 1 -- 172.21.15.89:0/2929058968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0017d50 con 0x555746be1f40 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90e2ffd700 1 -- 172.21.15.89:0/2929058968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90f000f040 con 0x555746be1f40 2023-08-25T02:27:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.571+0000 7f90e2ffd700 1 -- 172.21.15.89:0/2929058968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f001ac30 con 0x555746be1f40 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 -- 172.21.15.89:0/2929058968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 msgr2=0x555746be2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 0x7f90e80a26a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90f0021000 tx=0x7f90f001e000 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f90f0007820 tx=0x7f90f0007b30 comp rx=0 tx=0).stop 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 -- 172.21.15.89:0/2929058968 shutdown_connections 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bdcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be1f40 0x555746be2320 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 --2- 172.21.15.89:0/2929058968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746bdc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 -- 172.21.15.89:0/2929058968 >> 172.21.15.89:0/2929058968 conn(0x555746b51080 msgr2=0x555746bd6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 -- 172.21.15.89:0/2929058968 shutdown_connections 2023-08-25T02:27:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.572+0000 7f910b202b80 1 -- 172.21.15.89:0/2929058968 wait complete. 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 -- start start 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 Processor -- start 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746be8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746b2e930 con 0x555746be2860 2023-08-25T02:27:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746accb50 con 0x555746bdc990 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746acc850 con 0x555746be1f40 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f910b202b80 10 client.0 _ll_register_callbacks cb 0x555746a16a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48488/0 (socket says 172.21.15.89:48488) 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.573+0000 7f90fd5fb700 1 -- 172.21.15.89:0/4197588082 learned_addr learned my addr 172.21.15.89:0/4197588082 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90f7fff700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746be8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 -- 172.21.15.89:0/4197588082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 msgr2=0x555746be5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 -- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 msgr2=0x555746be8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746be8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 -- 172.21.15.89:0/4197588082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f0011040 con 0x555746be2860 2023-08-25T02:27:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746be8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90fd5fb700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f90f00095a0 tx=0x7f90f0009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24243f477edc898a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90d7fff700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0017ed0 con 0x555746be2860 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f90f0003c10 con 0x555746be2860 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555746bac160 con 0x555746be2860 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90f0003dd0 con 0x555746be2860 2023-08-25T02:27:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f910b202b80 10 client.15375 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.574+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55574695ad80 con 0x555746be2860 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555746bac160 con 0x555746be2860 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f90f00247d0 con 0x555746be2860 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 10 client.15375 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 10 client.15375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 20 client.15375 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746bf6280 0x555746bf86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 4 client.15375 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555746ad4b40 con 0x555746bf6280 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.575+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746bf6280 0x555746bf86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.576+0000 7f90d7fff700 10 client.15375 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.576+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746bf6280 0x555746bf86a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95bd55112b44f0b3 server_cookie=564521da504ff7c0 in_seq=0 out_seq=0 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f90d7fff700 1 -- 172.21.15.89:0/4197588082 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555746ad4b40 con 0x555746bf6280 2023-08-25T02:27:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f90d7fff700 10 client.15375 handle_command_reply: tid=0 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 client.15375 shutdown 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 msgr2=0x555746be6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f90f00095a0 tx=0x7f90f0009100 comp rx=0 tx=0).stop 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 shutdown_connections 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746bf6280 0x555746bf86a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746be8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be6000 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 --2- 172.21.15.89:0/4197588082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 >> 172.21.15.89:0/4197588082 conn(0x555746b51080 msgr2=0x555746bd1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f90d7fff700 0 client.15375 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 shutdown_connections 2023-08-25T02:27:12.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 1 -- 172.21.15.89:0/4197588082 wait complete. 2023-08-25T02:27:12.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.577+0000 7f910b202b80 20 client.15375 trim_cache size 0 max 16384 2023-08-25T02:27:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 Processor -- start 2023-08-25T02:27:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 -- start start 2023-08-25T02:27:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 0x555746bed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746be8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746bd72f0 con 0x555746bdc990 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557469b08c0 con 0x555746be1f40 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f910b202b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746bcf3f0 con 0x555746be2860 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48492/0 (socket says 172.21.15.89:48492) 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90f7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 0x555746bed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746be8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 -- 172.21.15.89:0/1734225601 learned_addr learned my addr 172.21.15.89:0/1734225601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 -- 172.21.15.89:0/1734225601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 msgr2=0x555746be8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746be8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 -- 172.21.15.89:0/1734225601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 msgr2=0x555746bed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 0x555746bed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90fcdfa700 1 -- 172.21.15.89:0/1734225601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555746943c00 con 0x555746bdc990 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.594+0000 7f90f7fff700 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 0x555746bed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005b40 tx=0x7f90ec0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a5c9d21034791c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f90e37fe700 1 -- 172.21.15.89:0/1734225601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec09a2b0 con 0x555746bdc990 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f90e37fe700 1 -- 172.21.15.89:0/1734225601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90ec005660 con 0x555746bdc990 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f90e37fe700 1 -- 172.21.15.89:0/1734225601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec097f50 con 0x555746bdc990 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 -- 172.21.15.89:0/1734225601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 msgr2=0x555746bdc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005b40 tx=0x7f90ec0cc040 comp rx=0 tx=0).stop 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 -- 172.21.15.89:0/1734225601 shutdown_connections 2023-08-25T02:27:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be2860 0x555746be8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746bdc990 0x555746bdc450 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.595+0000 7f910b202b80 1 --2- 172.21.15.89:0/1734225601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746be1f40 0x555746bed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- 172.21.15.89:0/1734225601 >> 172.21.15.89:0/1734225601 conn(0x555746b51080 msgr2=0x555746bdfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- 172.21.15.89:0/1734225601 shutdown_connections 2023-08-25T02:27:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- 172.21.15.89:0/1734225601 wait complete. 2023-08-25T02:27:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- start start 2023-08-25T02:27:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 Processor -- start 2023-08-25T02:27:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bed2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746be4a80 con 0x555746be2860 2023-08-25T02:27:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746be4bf0 con 0x555746bdc990 2023-08-25T02:27:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555746be4db0 con 0x555746be1f40 2023-08-25T02:27:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.596+0000 7f910b202b80 10 client.0 _ll_register_callbacks cb 0x7f90ec0092d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fd5fb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48502/0 (socket says 172.21.15.89:48502) 2023-08-25T02:27:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fd5fb700 1 -- 172.21.15.89:0/3631011733 learned_addr learned my addr 172.21.15.89:0/3631011733 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bed2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90f7fff700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 -- 172.21.15.89:0/3631011733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 msgr2=0x555746be3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 -- 172.21.15.89:0/3631011733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 msgr2=0x555746be43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 -- 172.21.15.89:0/3631011733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e4017e20 con 0x555746bdc990 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90fcdfa700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bed2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00fd50 tx=0x7f90ec095f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c121d27e24d317 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90d7fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec09d9e0 con 0x555746bdc990 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f90ec09a2b0 con 0x555746bdc990 2023-08-25T02:27:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555746bcf600 con 0x555746bdc990 2023-08-25T02:27:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90ec099bb0 con 0x555746bdc990 2023-08-25T02:27:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f910b202b80 10 client.25016 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f90f81a5b30 con 0x555746bdc990 2023-08-25T02:27:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.597+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90ec0c7080 con 0x555746bdc990 2023-08-25T02:27:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f90ec0a14a0 con 0x555746bdc990 2023-08-25T02:27:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 10 client.25016 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 10 client.25016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 20 client.25016 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746c4a360 0x555746c4c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 4 client.25016 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555746b27910 con 0x555746c4a360 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f90f7fff700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746c4a360 0x555746c4c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f90d7fff700 10 client.25016 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.598+0000 7f90f7fff700 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746c4a360 0x555746c4c750 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b47a5d0748fc80fc server_cookie=3928fea280278067 in_seq=0 out_seq=0 2023-08-25T02:27:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.613+0000 7f90d7fff700 1 -- 172.21.15.89:0/3631011733 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555746b27910 con 0x555746c4a360 2023-08-25T02:27:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.613+0000 7f90d7fff700 10 client.25016 handle_command_reply: tid=0 2023-08-25T02:27:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.613+0000 7f910b202b80 1 client.25016 shutdown 2023-08-25T02:27:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 msgr2=0x555746bed2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bed2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00fd50 tx=0x7f90ec095f30 comp rx=0 tx=0).stop 2023-08-25T02:27:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 shutdown_connections 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x555746c4a360 0x555746c4c750 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555746bdc990 0x555746bed2b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f90d7fff700 0 client.25016 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555746be2860 0x555746be43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 --2- 172.21.15.89:0/3631011733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555746be1f40 0x555746be3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 >> 172.21.15.89:0/3631011733 conn(0x555746b51080 msgr2=0x555746be2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 shutdown_connections 2023-08-25T02:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 1 -- 172.21.15.89:0/3631011733 wait complete. 2023-08-25T02:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.614+0000 7f910b202b80 20 client.25016 trim_cache size 0 max 16384 2023-08-25T02:27:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:27:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:27:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:27:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:27:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 msgr2=0x7f90e80a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 0x7f90e80a26a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90f0021000 tx=0x7f90f001e000 comp rx=0 tx=0).stop 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 msgr2=0x7f90f8072e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0095c0 tx=0x7f90ec00f710 comp rx=0 tx=0).stop 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 shutdown_connections 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f90e80a01e0 0x7f90e80a26a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90f81068d0 0x7f90f81a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90f81008c0 0x7f90f8072e00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 --2- 172.21.15.89:0/662664739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f8101280 0x7f90f8073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 >> 172.21.15.89:0/662664739 conn(0x7f90f80fc370 msgr2=0x7f90f80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 shutdown_connections 2023-08-25T02:27:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:12.615+0000 7f90ff05e700 1 -- 172.21.15.89:0/662664739 wait complete. 2023-08-25T02:27:12.666 INFO:teuthology.orchestra.run.smithi089.stderr:16500000 of 50000000 tuples (33%) done (elapsed 31.96 s, remaining 64.90 s) 2023-08-25T02:27:12.772 INFO:teuthology.orchestra.run.smithi089.stderr:16600000 of 50000000 tuples (33%) done (elapsed 32.07 s, remaining 64.53 s) 2023-08-25T02:27:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:16700000 of 50000000 tuples (33%) done (elapsed 32.17 s, remaining 64.14 s) 2023-08-25T02:27:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:16800000 of 50000000 tuples (33%) done (elapsed 32.29 s, remaining 63.82 s) 2023-08-25T02:27:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:16900000 of 50000000 tuples (33%) done (elapsed 32.37 s, remaining 63.40 s) 2023-08-25T02:27:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:17000000 of 50000000 tuples (34%) done (elapsed 32.48 s, remaining 63.05 s) 2023-08-25T02:27:13.299 INFO:teuthology.orchestra.run.smithi089.stderr:17100000 of 50000000 tuples (34%) done (elapsed 32.60 s, remaining 62.72 s) 2023-08-25T02:27:13.396 INFO:teuthology.orchestra.run.smithi089.stderr:17200000 of 50000000 tuples (34%) done (elapsed 32.69 s, remaining 62.35 s) 2023-08-25T02:27:13.496 INFO:teuthology.orchestra.run.smithi089.stderr:17300000 of 50000000 tuples (34%) done (elapsed 32.79 s, remaining 61.99 s) 2023-08-25T02:27:13.569 INFO:teuthology.orchestra.run.smithi089.stderr:17400000 of 50000000 tuples (34%) done (elapsed 32.87 s, remaining 61.58 s) 2023-08-25T02:27:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:17500000 of 50000000 tuples (35%) done (elapsed 33.01 s, remaining 61.30 s) 2023-08-25T02:27:13.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:13 smithi121 ceph-mon[100099]: pgmap v24: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 95 MiB/s wr, 605 op/s 2023-08-25T02:27:13.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:13 smithi122 ceph-mon[100457]: pgmap v24: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 95 MiB/s wr, 605 op/s 2023-08-25T02:27:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:17600000 of 50000000 tuples (35%) done (elapsed 33.09 s, remaining 60.92 s) 2023-08-25T02:27:13.836 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:13 smithi089 ceph-mon[98732]: pgmap v24: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 95 MiB/s wr, 605 op/s 2023-08-25T02:27:14.495 INFO:teuthology.orchestra.run.smithi089.stderr:17700000 of 50000000 tuples (35%) done (elapsed 33.79 s, remaining 61.67 s) 2023-08-25T02:27:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:17800000 of 50000000 tuples (35%) done (elapsed 33.89 s, remaining 61.31 s) 2023-08-25T02:27:14.703 INFO:teuthology.orchestra.run.smithi089.stderr:17900000 of 50000000 tuples (35%) done (elapsed 34.00 s, remaining 60.97 s) 2023-08-25T02:27:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:18000000 of 50000000 tuples (36%) done (elapsed 34.09 s, remaining 60.60 s) 2023-08-25T02:27:14.932 INFO:teuthology.orchestra.run.smithi089.stderr:18100000 of 50000000 tuples (36%) done (elapsed 34.23 s, remaining 60.33 s) 2023-08-25T02:27:15.039 INFO:teuthology.orchestra.run.smithi089.stderr:18200000 of 50000000 tuples (36%) done (elapsed 34.34 s, remaining 59.99 s) 2023-08-25T02:27:15.146 INFO:teuthology.orchestra.run.smithi089.stderr:18300000 of 50000000 tuples (36%) done (elapsed 34.44 s, remaining 59.67 s) 2023-08-25T02:27:15.260 INFO:teuthology.orchestra.run.smithi089.stderr:18400000 of 50000000 tuples (36%) done (elapsed 34.56 s, remaining 59.35 s) 2023-08-25T02:27:15.325 INFO:teuthology.orchestra.run.smithi089.stderr:18500000 of 50000000 tuples (37%) done (elapsed 34.62 s, remaining 58.95 s) 2023-08-25T02:27:15.449 INFO:teuthology.orchestra.run.smithi089.stderr:18600000 of 50000000 tuples (37%) done (elapsed 34.75 s, remaining 58.66 s) 2023-08-25T02:27:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:18700000 of 50000000 tuples (37%) done (elapsed 34.86 s, remaining 58.35 s) 2023-08-25T02:27:15.645 INFO:teuthology.orchestra.run.smithi089.stderr:18800000 of 50000000 tuples (37%) done (elapsed 34.94 s, remaining 57.99 s) 2023-08-25T02:27:15.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:15 smithi121 ceph-mon[100099]: pgmap v25: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 MiB/s wr, 424 op/s 2023-08-25T02:27:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:15 smithi122 ceph-mon[100457]: pgmap v25: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 MiB/s wr, 424 op/s 2023-08-25T02:27:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:18900000 of 50000000 tuples (37%) done (elapsed 35.07 s, remaining 57.71 s) 2023-08-25T02:27:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:15 smithi089 ceph-mon[98732]: pgmap v25: 97 pgs: 97 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 MiB/s wr, 424 op/s 2023-08-25T02:27:15.866 INFO:teuthology.orchestra.run.smithi089.stderr:19000000 of 50000000 tuples (38%) done (elapsed 35.16 s, remaining 57.37 s) 2023-08-25T02:27:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:19100000 of 50000000 tuples (38%) done (elapsed 35.27 s, remaining 57.05 s) 2023-08-25T02:27:16.080 INFO:teuthology.orchestra.run.smithi089.stderr:19200000 of 50000000 tuples (38%) done (elapsed 35.38 s, remaining 56.75 s) 2023-08-25T02:27:16.181 INFO:teuthology.orchestra.run.smithi089.stderr:19300000 of 50000000 tuples (38%) done (elapsed 35.48 s, remaining 56.44 s) 2023-08-25T02:27:16.273 INFO:teuthology.orchestra.run.smithi089.stderr:19400000 of 50000000 tuples (38%) done (elapsed 35.57 s, remaining 56.11 s) 2023-08-25T02:27:16.380 INFO:teuthology.orchestra.run.smithi089.stderr:19500000 of 50000000 tuples (39%) done (elapsed 35.68 s, remaining 55.80 s) 2023-08-25T02:27:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:19600000 of 50000000 tuples (39%) done (elapsed 35.80 s, remaining 55.52 s) 2023-08-25T02:27:16.599 INFO:teuthology.orchestra.run.smithi089.stderr:19700000 of 50000000 tuples (39%) done (elapsed 35.90 s, remaining 55.21 s) 2023-08-25T02:27:16.699 INFO:teuthology.orchestra.run.smithi089.stderr:19800000 of 50000000 tuples (39%) done (elapsed 36.00 s, remaining 54.90 s) 2023-08-25T02:27:16.775 INFO:teuthology.orchestra.run.smithi089.stderr:19900000 of 50000000 tuples (39%) done (elapsed 36.07 s, remaining 54.56 s) 2023-08-25T02:27:16.919 INFO:teuthology.orchestra.run.smithi089.stderr:20000000 of 50000000 tuples (40%) done (elapsed 36.22 s, remaining 54.33 s) 2023-08-25T02:27:16.984 INFO:teuthology.orchestra.run.smithi089.stderr:20100000 of 50000000 tuples (40%) done (elapsed 36.28 s, remaining 53.97 s) 2023-08-25T02:27:17.135 INFO:teuthology.orchestra.run.smithi089.stderr:20200000 of 50000000 tuples (40%) done (elapsed 36.43 s, remaining 53.75 s) 2023-08-25T02:27:17.232 INFO:teuthology.orchestra.run.smithi089.stderr:20300000 of 50000000 tuples (40%) done (elapsed 36.53 s, remaining 53.45 s) 2023-08-25T02:27:17.337 INFO:teuthology.orchestra.run.smithi089.stderr:20400000 of 50000000 tuples (40%) done (elapsed 36.63 s, remaining 53.16 s) 2023-08-25T02:27:17.412 INFO:teuthology.orchestra.run.smithi089.stderr:20500000 of 50000000 tuples (41%) done (elapsed 36.71 s, remaining 52.83 s) 2023-08-25T02:27:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:20600000 of 50000000 tuples (41%) done (elapsed 36.82 s, remaining 52.55 s) 2023-08-25T02:27:17.621 INFO:teuthology.orchestra.run.smithi089.stderr:20700000 of 50000000 tuples (41%) done (elapsed 36.92 s, remaining 52.26 s) 2023-08-25T02:27:17.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:17 smithi121 ceph-mon[100099]: pgmap v26: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 101 MiB/s wr, 468 op/s 2023-08-25T02:27:17.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:17 smithi122 ceph-mon[100457]: pgmap v26: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 101 MiB/s wr, 468 op/s 2023-08-25T02:27:17.737 INFO:teuthology.orchestra.run.smithi089.stderr:20800000 of 50000000 tuples (41%) done (elapsed 37.03 s, remaining 51.99 s) 2023-08-25T02:27:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:17 smithi089 ceph-mon[98732]: pgmap v26: 97 pgs: 97 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 101 MiB/s wr, 468 op/s 2023-08-25T02:27:17.846 INFO:teuthology.orchestra.run.smithi089.stderr:20900000 of 50000000 tuples (41%) done (elapsed 37.14 s, remaining 51.71 s) 2023-08-25T02:27:17.928 INFO:teuthology.orchestra.run.smithi089.stderr:21000000 of 50000000 tuples (42%) done (elapsed 37.23 s, remaining 51.41 s) 2023-08-25T02:27:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:21100000 of 50000000 tuples (42%) done (elapsed 37.35 s, remaining 51.15 s) 2023-08-25T02:27:18.145 INFO:teuthology.orchestra.run.smithi089.stderr:21200000 of 50000000 tuples (42%) done (elapsed 37.44 s, remaining 50.87 s) 2023-08-25T02:27:18.254 INFO:teuthology.orchestra.run.smithi089.stderr:21300000 of 50000000 tuples (42%) done (elapsed 37.55 s, remaining 50.60 s) 2023-08-25T02:27:18.349 INFO:teuthology.orchestra.run.smithi089.stderr:21400000 of 50000000 tuples (42%) done (elapsed 37.65 s, remaining 50.31 s) 2023-08-25T02:27:18.515 INFO:teuthology.orchestra.run.smithi089.stderr:21500000 of 50000000 tuples (43%) done (elapsed 37.81 s, remaining 50.12 s) 2023-08-25T02:27:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:21600000 of 50000000 tuples (43%) done (elapsed 38.41 s, remaining 50.50 s) 2023-08-25T02:27:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:21700000 of 50000000 tuples (43%) done (elapsed 38.52 s, remaining 50.24 s) 2023-08-25T02:27:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:21800000 of 50000000 tuples (43%) done (elapsed 38.62 s, remaining 49.96 s) 2023-08-25T02:27:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:21900000 of 50000000 tuples (43%) done (elapsed 38.72 s, remaining 49.68 s) 2023-08-25T02:27:19.536 INFO:teuthology.orchestra.run.smithi089.stderr:22000000 of 50000000 tuples (44%) done (elapsed 38.83 s, remaining 49.42 s) 2023-08-25T02:27:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:22100000 of 50000000 tuples (44%) done (elapsed 38.95 s, remaining 49.17 s) 2023-08-25T02:27:19.755 INFO:teuthology.orchestra.run.smithi089.stderr:22200000 of 50000000 tuples (44%) done (elapsed 39.05 s, remaining 48.90 s) 2023-08-25T02:27:19.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:19 smithi089 ceph-mon[98732]: pgmap v27: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 143 MiB/s wr, 196 op/s 2023-08-25T02:27:19.859 INFO:teuthology.orchestra.run.smithi089.stderr:22300000 of 50000000 tuples (44%) done (elapsed 39.16 s, remaining 48.64 s) 2023-08-25T02:27:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:22400000 of 50000000 tuples (44%) done (elapsed 39.23 s, remaining 48.33 s) 2023-08-25T02:27:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:19 smithi121 ceph-mon[100099]: pgmap v27: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 143 MiB/s wr, 196 op/s 2023-08-25T02:27:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:19 smithi122 ceph-mon[100457]: pgmap v27: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 143 MiB/s wr, 196 op/s 2023-08-25T02:27:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:22500000 of 50000000 tuples (45%) done (elapsed 39.34 s, remaining 48.08 s) 2023-08-25T02:27:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:22600000 of 50000000 tuples (45%) done (elapsed 39.44 s, remaining 47.82 s) 2023-08-25T02:27:20.257 INFO:teuthology.orchestra.run.smithi089.stderr:22700000 of 50000000 tuples (45%) done (elapsed 39.55 s, remaining 47.57 s) 2023-08-25T02:27:20.364 INFO:teuthology.orchestra.run.smithi089.stderr:22800000 of 50000000 tuples (45%) done (elapsed 39.66 s, remaining 47.32 s) 2023-08-25T02:27:20.455 INFO:teuthology.orchestra.run.smithi089.stderr:22900000 of 50000000 tuples (45%) done (elapsed 39.75 s, remaining 47.04 s) 2023-08-25T02:27:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:23000000 of 50000000 tuples (46%) done (elapsed 39.86 s, remaining 46.79 s) 2023-08-25T02:27:20.670 INFO:teuthology.orchestra.run.smithi089.stderr:23100000 of 50000000 tuples (46%) done (elapsed 39.97 s, remaining 46.54 s) 2023-08-25T02:27:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:23200000 of 50000000 tuples (46%) done (elapsed 40.07 s, remaining 46.29 s) 2023-08-25T02:27:20.878 INFO:teuthology.orchestra.run.smithi089.stderr:23300000 of 50000000 tuples (46%) done (elapsed 40.18 s, remaining 46.04 s) 2023-08-25T02:27:20.982 INFO:teuthology.orchestra.run.smithi089.stderr:23400000 of 50000000 tuples (46%) done (elapsed 40.28 s, remaining 45.79 s) 2023-08-25T02:27:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:23500000 of 50000000 tuples (47%) done (elapsed 40.35 s, remaining 45.50 s) 2023-08-25T02:27:21.168 INFO:teuthology.orchestra.run.smithi089.stderr:23600000 of 50000000 tuples (47%) done (elapsed 40.47 s, remaining 45.27 s) 2023-08-25T02:27:21.271 INFO:teuthology.orchestra.run.smithi089.stderr:23700000 of 50000000 tuples (47%) done (elapsed 40.57 s, remaining 45.02 s) 2023-08-25T02:27:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:23800000 of 50000000 tuples (47%) done (elapsed 40.67 s, remaining 44.77 s) 2023-08-25T02:27:21.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:21 smithi121 ceph-mon[100099]: pgmap v28: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 142 MiB/s wr, 192 op/s 2023-08-25T02:27:21.476 INFO:teuthology.orchestra.run.smithi089.stderr:23900000 of 50000000 tuples (47%) done (elapsed 40.77 s, remaining 44.53 s) 2023-08-25T02:27:21.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:21 smithi122 ceph-mon[100457]: pgmap v28: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 142 MiB/s wr, 192 op/s 2023-08-25T02:27:21.574 INFO:teuthology.orchestra.run.smithi089.stderr:24000000 of 50000000 tuples (48%) done (elapsed 40.87 s, remaining 44.28 s) 2023-08-25T02:27:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:21 smithi089 ceph-mon[98732]: pgmap v28: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 142 MiB/s wr, 192 op/s 2023-08-25T02:27:21.662 INFO:teuthology.orchestra.run.smithi089.stderr:24100000 of 50000000 tuples (48%) done (elapsed 40.96 s, remaining 44.02 s) 2023-08-25T02:27:21.781 INFO:teuthology.orchestra.run.smithi089.stderr:24200000 of 50000000 tuples (48%) done (elapsed 41.08 s, remaining 43.79 s) 2023-08-25T02:27:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:24300000 of 50000000 tuples (48%) done (elapsed 41.18 s, remaining 43.56 s) 2023-08-25T02:27:21.966 INFO:teuthology.orchestra.run.smithi089.stderr:24400000 of 50000000 tuples (48%) done (elapsed 41.26 s, remaining 43.29 s) 2023-08-25T02:27:22.067 INFO:teuthology.orchestra.run.smithi089.stderr:24500000 of 50000000 tuples (49%) done (elapsed 41.37 s, remaining 43.05 s) 2023-08-25T02:27:22.209 INFO:teuthology.orchestra.run.smithi089.stderr:24600000 of 50000000 tuples (49%) done (elapsed 41.51 s, remaining 42.86 s) 2023-08-25T02:27:22.288 INFO:teuthology.orchestra.run.smithi089.stderr:24700000 of 50000000 tuples (49%) done (elapsed 41.59 s, remaining 42.60 s) 2023-08-25T02:27:22.385 INFO:teuthology.orchestra.run.smithi089.stderr:24800000 of 50000000 tuples (49%) done (elapsed 41.68 s, remaining 42.35 s) 2023-08-25T02:27:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:24900000 of 50000000 tuples (49%) done (elapsed 42.01 s, remaining 42.35 s) 2023-08-25T02:27:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:25000000 of 50000000 tuples (50%) done (elapsed 42.42 s, remaining 42.42 s) 2023-08-25T02:27:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:25100000 of 50000000 tuples (50%) done (elapsed 42.52 s, remaining 42.18 s) 2023-08-25T02:27:23.328 INFO:teuthology.orchestra.run.smithi089.stderr:25200000 of 50000000 tuples (50%) done (elapsed 42.63 s, remaining 41.95 s) 2023-08-25T02:27:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:25300000 of 50000000 tuples (50%) done (elapsed 42.74 s, remaining 41.72 s) 2023-08-25T02:27:23.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:23 smithi121 ceph-mon[100099]: pgmap v29: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 221 MiB/s wr, 229 op/s 2023-08-25T02:27:23.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:23 smithi122 ceph-mon[100457]: pgmap v29: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 221 MiB/s wr, 229 op/s 2023-08-25T02:27:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:23 smithi089 ceph-mon[98732]: pgmap v29: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 221 MiB/s wr, 229 op/s 2023-08-25T02:27:23.912 INFO:teuthology.orchestra.run.smithi089.stderr:25400000 of 50000000 tuples (50%) done (elapsed 43.21 s, remaining 41.85 s) 2023-08-25T02:27:24.231 INFO:teuthology.orchestra.run.smithi089.stderr:25500000 of 50000000 tuples (51%) done (elapsed 43.53 s, remaining 41.82 s) 2023-08-25T02:27:24.358 INFO:teuthology.orchestra.run.smithi089.stderr:25600000 of 50000000 tuples (51%) done (elapsed 43.66 s, remaining 41.61 s) 2023-08-25T02:27:24.460 INFO:teuthology.orchestra.run.smithi089.stderr:25700000 of 50000000 tuples (51%) done (elapsed 43.76 s, remaining 41.37 s) 2023-08-25T02:27:24.551 INFO:teuthology.orchestra.run.smithi089.stderr:25800000 of 50000000 tuples (51%) done (elapsed 43.85 s, remaining 41.13 s) 2023-08-25T02:27:24.673 INFO:teuthology.orchestra.run.smithi089.stderr:25900000 of 50000000 tuples (51%) done (elapsed 43.97 s, remaining 40.91 s) 2023-08-25T02:27:24.743 INFO:teuthology.orchestra.run.smithi089.stderr:26000000 of 50000000 tuples (52%) done (elapsed 44.04 s, remaining 40.65 s) 2023-08-25T02:27:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:26100000 of 50000000 tuples (52%) done (elapsed 44.16 s, remaining 40.44 s) 2023-08-25T02:27:24.991 INFO:teuthology.orchestra.run.smithi089.stderr:26200000 of 50000000 tuples (52%) done (elapsed 44.29 s, remaining 40.23 s) 2023-08-25T02:27:25.056 INFO:teuthology.orchestra.run.smithi089.stderr:26300000 of 50000000 tuples (52%) done (elapsed 44.35 s, remaining 39.97 s) 2023-08-25T02:27:25.169 INFO:teuthology.orchestra.run.smithi089.stderr:26400000 of 50000000 tuples (52%) done (elapsed 44.47 s, remaining 39.75 s) 2023-08-25T02:27:25.279 INFO:teuthology.orchestra.run.smithi089.stderr:26500000 of 50000000 tuples (53%) done (elapsed 44.58 s, remaining 39.53 s) 2023-08-25T02:27:25.567 INFO:teuthology.orchestra.run.smithi089.stderr:26600000 of 50000000 tuples (53%) done (elapsed 44.87 s, remaining 39.47 s) 2023-08-25T02:27:25.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:25 smithi121 ceph-mon[100099]: pgmap v30: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 152 MiB/s wr, 121 op/s 2023-08-25T02:27:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:25 smithi122 ceph-mon[100457]: pgmap v30: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 152 MiB/s wr, 121 op/s 2023-08-25T02:27:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 -- 172.21.15.89:0/3122564120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57100ffeb0 msgr2=0x7f5710100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 --2- 172.21.15.89:0/3122564120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57100ffeb0 0x7f5710100330 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5704009990 tx=0x7f5704009ca0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 -- 172.21.15.89:0/3122564120 shutdown_connections 2023-08-25T02:27:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 --2- 172.21.15.89:0/3122564120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5710105800 0x7f5710105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 --2- 172.21.15.89:0/3122564120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710100870 0x7f571010c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 --2- 172.21.15.89:0/3122564120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57100ffeb0 0x7f5710100330 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 -- 172.21.15.89:0/3122564120 >> 172.21.15.89:0/3122564120 conn(0x7f5710075ce0 msgr2=0x7f57100fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 -- 172.21.15.89:0/3122564120 shutdown_connections 2023-08-25T02:27:25.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.736+0000 7f5715b02700 1 -- 172.21.15.89:0/3122564120 wait complete. 2023-08-25T02:27:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 Processor -- start 2023-08-25T02:27:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 -- start start 2023-08-25T02:27:25.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 0x7f57101979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710105800 0x7f57101a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101a2e60 con 0x7f5710100870 2023-08-25T02:27:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101a2fd0 con 0x7f57100ffeb0 2023-08-25T02:27:25.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f5715b02700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57101a3190 con 0x7f5710105800 2023-08-25T02:27:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f570effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f570effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48514/0 (socket says 172.21.15.89:48514) 2023-08-25T02:27:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 0x7f57101979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.737+0000 7f570effd700 1 -- 172.21.15.89:0/1504171501 learned_addr learned my addr 172.21.15.89:0/1504171501 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570ffff700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710105800 0x7f57101a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 -- 172.21.15.89:0/1504171501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710105800 msgr2=0x7f57101a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710105800 0x7f57101a2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 -- 172.21.15.89:0/1504171501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 msgr2=0x7f57101979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 0x7f57101979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 -- 172.21.15.89:0/1504171501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040095f0 con 0x7f5710100870 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570f7fe700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 0x7f57101979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570effd700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5704000c00 tx=0x7f5704004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570cff9700 1 -- 172.21.15.89:0/1504171501 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570401d070 con 0x7f5710100870 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57101a3410 con 0x7f5710100870 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57101a3910 con 0x7f5710100870 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570cff9700 1 -- 172.21.15.89:0/1504171501 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5704021410 con 0x7f5710100870 2023-08-25T02:27:25.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.738+0000 7f570cff9700 1 -- 172.21.15.89:0/1504171501 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704017650 con 0x7f5710100870 2023-08-25T02:27:25.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.740+0000 7f570cff9700 1 -- 172.21.15.89:0/1504171501 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 134904+0+0 (secure 0 0 0) 0x7f5704021580 con 0x7f5710100870 2023-08-25T02:27:25.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.741+0000 7f570cff9700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 0x7f56f80a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.741+0000 7f570f7fe700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 0x7f56f80a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.741+0000 7f570cff9700 1 -- 172.21.15.89:0/1504171501 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f57040ccc10 con 0x7f5710100870 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.741+0000 7f570f7fe700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 0x7f56f80a26a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5700009fd0 tx=0x7f5700009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.742+0000 7f5721ca6b80 1 Processor -- start 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.742+0000 7f5721ca6b80 1 -- start start 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.742+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e661320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e5c3750 con 0x56345e65b990 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e5b4720 con 0x56345e661860 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e5b2040 con 0x56345e660f40 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e661320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:38932/0 (socket says 172.21.15.89:38932) 2023-08-25T02:27:25.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 -- 172.21.15.89:0/621262061 learned_addr learned my addr 172.21.15.89:0/621262061 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:25.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570ffff700 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 -- 172.21.15.89:0/621262061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 msgr2=0x56345e661320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e661320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 -- 172.21.15.89:0/621262061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 msgr2=0x56345e65bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 -- 172.21.15.89:0/621262061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56345e42d5d0 con 0x56345e661860 2023-08-25T02:27:25.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570f7fe700 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e661320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:25.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f570effd700 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5704096bd0 tx=0x7f570409a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2a07015bbd2f0b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f56f5ffb700 1 -- 172.21.15.89:0/621262061 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570409a160 con 0x56345e661860 2023-08-25T02:27:25.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f56f5ffb700 1 -- 172.21.15.89:0/621262061 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f570409df90 con 0x56345e661860 2023-08-25T02:27:25.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.743+0000 7f56f5ffb700 1 -- 172.21.15.89:0/621262061 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704099650 con 0x56345e661860 2023-08-25T02:27:25.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 -- 172.21.15.89:0/621262061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 msgr2=0x56345e65b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5704096bd0 tx=0x7f570409a030 comp rx=0 tx=0).stop 2023-08-25T02:27:25.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 -- 172.21.15.89:0/621262061 shutdown_connections 2023-08-25T02:27:25.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e661320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/621262061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e65b450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.744+0000 7f5721ca6b80 1 -- 172.21.15.89:0/621262061 >> 172.21.15.89:0/621262061 conn(0x56345e5d0080 msgr2=0x56345e655d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 -- 172.21.15.89:0/621262061 shutdown_connections 2023-08-25T02:27:25.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 -- 172.21.15.89:0/621262061 wait complete. 2023-08-25T02:27:25.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 -- start start 2023-08-25T02:27:25.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 Processor -- start 2023-08-25T02:27:25.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e664940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e665080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e5ad930 con 0x56345e661860 2023-08-25T02:27:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.745+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e54bb50 con 0x56345e65b990 2023-08-25T02:27:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e54b850 con 0x56345e660f40 2023-08-25T02:27:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f5721ca6b80 10 client.0 _ll_register_callbacks cb 0x56345e495a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e665080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:38936/0 (socket says 172.21.15.89:38936) 2023-08-25T02:27:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 -- 172.21.15.89:0/983111239 learned_addr learned my addr 172.21.15.89:0/983111239 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570ffff700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e665080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48530/0 (socket says 172.21.15.89:48530) 2023-08-25T02:27:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570effd700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e664940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 -- 172.21.15.89:0/983111239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 msgr2=0x56345e664940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e664940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 -- 172.21.15.89:0/983111239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 msgr2=0x56345e665080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e665080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 -- 172.21.15.89:0/983111239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5700017070 con 0x56345e65b990 2023-08-25T02:27:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f570f7fe700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f570000ebb0 tx=0x7f57000047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b48da48aae73be server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f56eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700004460 con 0x56345e65b990 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56345e62b160 con 0x56345e65b990 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f570001f810 con 0x56345e65b990 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.746+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f570001f9d0 con 0x56345e65b990 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 10 client.25028 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:25.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f57101a3910 con 0x56345e65b990 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56345e62b160 con 0x56345e65b990 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f5700031480 con 0x56345e65b990 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 10 client.25028 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 10 client.25028 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 20 client.25028 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:25.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e67d680 0x56345e67faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 4 client.25028 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:27:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56345e553b40 con 0x56345e67d680 2023-08-25T02:27:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.747+0000 7f570effd700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e67d680 0x56345e67faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f56eeffd700 10 client.25028 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:25.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f570effd700 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e67d680 0x56345e67faa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7b40d765cf2a371 server_cookie=5814d4ec1efcb7d in_seq=0 out_seq=0 2023-08-25T02:27:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f56eeffd700 1 -- 172.21.15.89:0/983111239 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56345e553b40 con 0x56345e67d680 2023-08-25T02:27:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f56eeffd700 10 client.25028 handle_command_reply: tid=0 2023-08-25T02:27:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 client.25028 shutdown 2023-08-25T02:27:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 msgr2=0x56345e6676c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f570000ebb0 tx=0x7f57000047c0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 shutdown_connections 2023-08-25T02:27:25.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e67d680 0x56345e67faa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e65b990 0x56345e6676c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e665080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/983111239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e664940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f56eeffd700 0 client.25028 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 >> 172.21.15.89:0/983111239 conn(0x56345e5d0080 msgr2=0x56345e650820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 shutdown_connections 2023-08-25T02:27:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.748+0000 7f5721ca6b80 1 -- 172.21.15.89:0/983111239 wait complete. 2023-08-25T02:27:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.749+0000 7f5721ca6b80 20 client.25028 trim_cache size 0 max 16384 2023-08-25T02:27:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 Processor -- start 2023-08-25T02:27:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 -- start start 2023-08-25T02:27:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e66bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e6671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e42f8c0 con 0x56345e65b990 2023-08-25T02:27:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e656c50 con 0x56345e661860 2023-08-25T02:27:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e661320 con 0x56345e660f40 2023-08-25T02:27:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.768+0000 7f570f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48534/0 (socket says 172.21.15.89:48534) 2023-08-25T02:27:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 -- 172.21.15.89:0/4218777551 learned_addr learned my addr 172.21.15.89:0/4218777551 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570effd700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e66bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 -- 172.21.15.89:0/4218777551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 msgr2=0x56345e66bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e66bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 -- 172.21.15.89:0/4218777551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 msgr2=0x56345e6671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570ffff700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e6671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e6671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 -- 172.21.15.89:0/4218777551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56345e3d9d80 con 0x56345e65b990 2023-08-25T02:27:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570effd700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e66bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:27:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f570f7fe700 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5700013010 tx=0x7f5700011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42c034cba6c7353 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f56f67fc700 1 -- 172.21.15.89:0/4218777551 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700003ed0 con 0x56345e65b990 2023-08-25T02:27:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f56f67fc700 1 -- 172.21.15.89:0/4218777551 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5700031a00 con 0x56345e65b990 2023-08-25T02:27:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.769+0000 7f56f67fc700 1 -- 172.21.15.89:0/4218777551 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570001f6e0 con 0x56345e65b990 2023-08-25T02:27:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 -- 172.21.15.89:0/4218777551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 msgr2=0x56345e65b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5700013010 tx=0x7f5700011c10 comp rx=0 tx=0).stop 2023-08-25T02:27:25.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 -- 172.21.15.89:0/4218777551 shutdown_connections 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e661860 0x56345e6671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e65b990 0x56345e65b450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/4218777551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e660f40 0x56345e66bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 -- 172.21.15.89:0/4218777551 >> 172.21.15.89:0/4218777551 conn(0x56345e5d0080 msgr2=0x56345e65ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.770+0000 7f5721ca6b80 1 -- 172.21.15.89:0/4218777551 shutdown_connections 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.771+0000 7f5721ca6b80 1 -- 172.21.15.89:0/4218777551 wait complete. 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.771+0000 7f5721ca6b80 1 -- start start 2023-08-25T02:27:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.771+0000 7f5721ca6b80 1 Processor -- start 2023-08-25T02:27:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.771+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e65b990 0x56345e65deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e660f40 0x56345e65c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e65e570 con 0x56345e661860 2023-08-25T02:27:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e65d3d0 con 0x56345e660f40 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56345e65d590 con 0x56345e65b990 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 10 client.0 _ll_register_callbacks cb 0x56345e495a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48542/0 (socket says 172.21.15.89:48542) 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 -- 172.21.15.89:0/1390894334 learned_addr learned my addr 172.21.15.89:0/1390894334 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:25.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 -- 172.21.15.89:0/1390894334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e65b990 msgr2=0x56345e65deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e65b990 0x56345e65deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 -- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e660f40 msgr2=0x56345e65c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e660f40 0x56345e65c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 -- 172.21.15.89:0/1390894334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57101a3910 con 0x56345e661860 2023-08-25T02:27:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f570ffff700 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5708007e70 tx=0x7f57080145a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07eefa7595773af server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f56eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:27:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570800e030 con 0x56345e661860 2023-08-25T02:27:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56345e64e600 con 0x56345e661860 2023-08-25T02:27:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f570800f070 con 0x56345e661860 2023-08-25T02:27:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5708014cd0 con 0x56345e661860 2023-08-25T02:27:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 10 client.15399 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.772+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56345e3c2c00 con 0x56345e661860 2023-08-25T02:27:25.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56345e64e600 con 0x56345e661860 2023-08-25T02:27:25.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f57080203b0 con 0x56345e661860 2023-08-25T02:27:25.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 10 client.15399 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:25.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 10 client.15399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:25.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 20 client.15399 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:25.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e6d2720 0x56345e6d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:25.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 4 client.15399 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:27:25.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56345e5a6910 con 0x56345e6d2720 2023-08-25T02:27:25.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f570f7fe700 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e6d2720 0x56345e6d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:25.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f56eeffd700 10 client.15399 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:25.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.773+0000 7f570f7fe700 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e6d2720 0x56345e6d4b60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f06946355d83ded server_cookie=322d7c3c47b995c2 in_seq=0 out_seq=0 2023-08-25T02:27:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f56eeffd700 1 -- 172.21.15.89:0/1390894334 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56345e5a6910 con 0x56345e6d2720 2023-08-25T02:27:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f56eeffd700 10 client.15399 handle_command_reply: tid=0 2023-08-25T02:27:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 client.15399 shutdown 2023-08-25T02:27:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 msgr2=0x56345e65ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5708007e70 tx=0x7f57080145a0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 shutdown_connections 2023-08-25T02:27:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56345e6d2720 0x56345e6d4b60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56345e661860 0x56345e65ce90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56345e65b990 0x56345e65deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f56eeffd700 0 client.15399 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 --2- 172.21.15.89:0/1390894334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56345e660f40 0x56345e65c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 >> 172.21.15.89:0/1390894334 conn(0x56345e5d0080 msgr2=0x56345e661c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 shutdown_connections 2023-08-25T02:27:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 1 -- 172.21.15.89:0/1390894334 wait complete. 2023-08-25T02:27:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.793+0000 7f5721ca6b80 20 client.15399 trim_cache size 0 max 16384 2023-08-25T02:27:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:27:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:27:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:27:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:27:25.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 msgr2=0x7f56f80a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 0x7f56f80a26a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5700009fd0 tx=0x7f5700009260 comp rx=0 tx=0).stop 2023-08-25T02:27:25.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 msgr2=0x7f5710197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:25.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5704000c00 tx=0x7f5704004790 comp rx=0 tx=0).stop 2023-08-25T02:27:25.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 shutdown_connections 2023-08-25T02:27:25.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f80a01e0 0x7f56f80a26a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57100ffeb0 0x7f57101979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5710105800 0x7f57101a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 --2- 172.21.15.89:0/1504171501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5710100870 0x7f5710197f30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 >> 172.21.15.89:0/1504171501 conn(0x7f5710075ce0 msgr2=0x7f57100fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 shutdown_connections 2023-08-25T02:27:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:25.794+0000 7f5715b02700 1 -- 172.21.15.89:0/1504171501 wait complete. 2023-08-25T02:27:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:25 smithi089 ceph-mon[98732]: pgmap v30: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 152 MiB/s wr, 121 op/s 2023-08-25T02:27:26.017 INFO:teuthology.orchestra.run.smithi089.stderr:26700000 of 50000000 tuples (53%) done (elapsed 45.31 s, remaining 39.54 s) 2023-08-25T02:27:26.124 INFO:teuthology.orchestra.run.smithi089.stderr:26800000 of 50000000 tuples (53%) done (elapsed 45.42 s, remaining 39.32 s) 2023-08-25T02:27:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:26900000 of 50000000 tuples (53%) done (elapsed 45.64 s, remaining 39.20 s) 2023-08-25T02:27:26.870 INFO:teuthology.orchestra.run.smithi089.stderr:27000000 of 50000000 tuples (54%) done (elapsed 46.17 s, remaining 39.33 s) 2023-08-25T02:27:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:27100000 of 50000000 tuples (54%) done (elapsed 46.28 s, remaining 39.11 s) 2023-08-25T02:27:27.061 INFO:teuthology.orchestra.run.smithi089.stderr:27200000 of 50000000 tuples (54%) done (elapsed 46.36 s, remaining 38.86 s) 2023-08-25T02:27:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:27300000 of 50000000 tuples (54%) done (elapsed 46.46 s, remaining 38.63 s) 2023-08-25T02:27:27.273 INFO:teuthology.orchestra.run.smithi089.stderr:27400000 of 50000000 tuples (54%) done (elapsed 46.57 s, remaining 38.41 s) 2023-08-25T02:27:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:27500000 of 50000000 tuples (55%) done (elapsed 46.66 s, remaining 38.18 s) 2023-08-25T02:27:27.475 INFO:teuthology.orchestra.run.smithi089.stderr:27600000 of 50000000 tuples (55%) done (elapsed 46.77 s, remaining 37.96 s) 2023-08-25T02:27:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:27700000 of 50000000 tuples (55%) done (elapsed 46.85 s, remaining 37.71 s) 2023-08-25T02:27:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:27800000 of 50000000 tuples (55%) done (elapsed 46.98 s, remaining 37.52 s) 2023-08-25T02:27:27.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:27 smithi121 ceph-mon[100099]: pgmap v31: 97 pgs: 97 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 179 MiB/s wr, 138 op/s 2023-08-25T02:27:27.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:27 smithi122 ceph-mon[100457]: pgmap v31: 97 pgs: 97 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 179 MiB/s wr, 138 op/s 2023-08-25T02:27:27.748 INFO:teuthology.orchestra.run.smithi089.stderr:27900000 of 50000000 tuples (55%) done (elapsed 47.05 s, remaining 37.27 s) 2023-08-25T02:27:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:27 smithi089 ceph-mon[98732]: pgmap v31: 97 pgs: 97 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 179 MiB/s wr, 138 op/s 2023-08-25T02:27:27.847 INFO:teuthology.orchestra.run.smithi089.stderr:28000000 of 50000000 tuples (56%) done (elapsed 47.15 s, remaining 37.04 s) 2023-08-25T02:27:27.956 INFO:teuthology.orchestra.run.smithi089.stderr:28100000 of 50000000 tuples (56%) done (elapsed 47.25 s, remaining 36.83 s) 2023-08-25T02:27:28.058 INFO:teuthology.orchestra.run.smithi089.stderr:28200000 of 50000000 tuples (56%) done (elapsed 47.36 s, remaining 36.61 s) 2023-08-25T02:27:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:28300000 of 50000000 tuples (56%) done (elapsed 47.45 s, remaining 36.38 s) 2023-08-25T02:27:28.256 INFO:teuthology.orchestra.run.smithi089.stderr:28400000 of 50000000 tuples (56%) done (elapsed 47.55 s, remaining 36.17 s) 2023-08-25T02:27:28.366 INFO:teuthology.orchestra.run.smithi089.stderr:28500000 of 50000000 tuples (57%) done (elapsed 47.66 s, remaining 35.96 s) 2023-08-25T02:27:28.453 INFO:teuthology.orchestra.run.smithi089.stderr:28600000 of 50000000 tuples (57%) done (elapsed 47.75 s, remaining 35.73 s) 2023-08-25T02:27:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:28700000 of 50000000 tuples (57%) done (elapsed 47.85 s, remaining 35.52 s) 2023-08-25T02:27:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:28800000 of 50000000 tuples (57%) done (elapsed 47.94 s, remaining 35.29 s) 2023-08-25T02:27:28.768 INFO:teuthology.orchestra.run.smithi089.stderr:28900000 of 50000000 tuples (57%) done (elapsed 48.07 s, remaining 35.09 s) 2023-08-25T02:27:28.855 INFO:teuthology.orchestra.run.smithi089.stderr:29000000 of 50000000 tuples (58%) done (elapsed 48.15 s, remaining 34.87 s) 2023-08-25T02:27:28.961 INFO:teuthology.orchestra.run.smithi089.stderr:29100000 of 50000000 tuples (58%) done (elapsed 48.26 s, remaining 34.66 s) 2023-08-25T02:27:29.072 INFO:teuthology.orchestra.run.smithi089.stderr:29200000 of 50000000 tuples (58%) done (elapsed 48.37 s, remaining 34.46 s) 2023-08-25T02:27:29.173 INFO:teuthology.orchestra.run.smithi089.stderr:29300000 of 50000000 tuples (58%) done (elapsed 48.47 s, remaining 34.24 s) 2023-08-25T02:27:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:29 smithi089 ceph-mon[98732]: pgmap v32: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 192 MiB/s wr, 204 op/s 2023-08-25T02:27:29.886 INFO:teuthology.orchestra.run.smithi089.stderr:29400000 of 50000000 tuples (58%) done (elapsed 49.18 s, remaining 34.46 s) 2023-08-25T02:27:29.955 INFO:teuthology.orchestra.run.smithi089.stderr:29500000 of 50000000 tuples (59%) done (elapsed 49.25 s, remaining 34.23 s) 2023-08-25T02:27:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:29 smithi121 ceph-mon[100099]: pgmap v32: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 192 MiB/s wr, 204 op/s 2023-08-25T02:27:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:29 smithi122 ceph-mon[100457]: pgmap v32: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 192 MiB/s wr, 204 op/s 2023-08-25T02:27:30.094 INFO:teuthology.orchestra.run.smithi089.stderr:29600000 of 50000000 tuples (59%) done (elapsed 49.39 s, remaining 34.04 s) 2023-08-25T02:27:30.178 INFO:teuthology.orchestra.run.smithi089.stderr:29700000 of 50000000 tuples (59%) done (elapsed 49.48 s, remaining 33.82 s) 2023-08-25T02:27:30.284 INFO:teuthology.orchestra.run.smithi089.stderr:29800000 of 50000000 tuples (59%) done (elapsed 49.58 s, remaining 33.61 s) 2023-08-25T02:27:30.373 INFO:teuthology.orchestra.run.smithi089.stderr:29900000 of 50000000 tuples (59%) done (elapsed 49.67 s, remaining 33.39 s) 2023-08-25T02:27:30.494 INFO:teuthology.orchestra.run.smithi089.stderr:30000000 of 50000000 tuples (60%) done (elapsed 49.79 s, remaining 33.19 s) 2023-08-25T02:27:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:30100000 of 50000000 tuples (60%) done (elapsed 49.89 s, remaining 32.98 s) 2023-08-25T02:27:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:30200000 of 50000000 tuples (60%) done (elapsed 49.98 s, remaining 32.77 s) 2023-08-25T02:27:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:30300000 of 50000000 tuples (60%) done (elapsed 50.09 s, remaining 32.57 s) 2023-08-25T02:27:30.892 INFO:teuthology.orchestra.run.smithi089.stderr:30400000 of 50000000 tuples (60%) done (elapsed 50.19 s, remaining 32.36 s) 2023-08-25T02:27:30.992 INFO:teuthology.orchestra.run.smithi089.stderr:30500000 of 50000000 tuples (61%) done (elapsed 50.29 s, remaining 32.15 s) 2023-08-25T02:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:30600000 of 50000000 tuples (61%) done (elapsed 50.39 s, remaining 31.95 s) 2023-08-25T02:27:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:31 smithi121 ceph-mon[100099]: pgmap v33: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 149 MiB/s wr, 163 op/s 2023-08-25T02:27:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:31 smithi122 ceph-mon[100457]: pgmap v33: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 149 MiB/s wr, 163 op/s 2023-08-25T02:27:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:31 smithi089 ceph-mon[98732]: pgmap v33: 97 pgs: 97 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 149 MiB/s wr, 163 op/s 2023-08-25T02:27:31.612 INFO:teuthology.orchestra.run.smithi089.stderr:30700000 of 50000000 tuples (61%) done (elapsed 50.91 s, remaining 32.01 s) 2023-08-25T02:27:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:30800000 of 50000000 tuples (61%) done (elapsed 51.17 s, remaining 31.90 s) 2023-08-25T02:27:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:30900000 of 50000000 tuples (61%) done (elapsed 51.27 s, remaining 31.69 s) 2023-08-25T02:27:32.078 INFO:teuthology.orchestra.run.smithi089.stderr:31000000 of 50000000 tuples (62%) done (elapsed 51.38 s, remaining 31.49 s) 2023-08-25T02:27:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:31100000 of 50000000 tuples (62%) done (elapsed 51.47 s, remaining 31.28 s) 2023-08-25T02:27:32.274 INFO:teuthology.orchestra.run.smithi089.stderr:31200000 of 50000000 tuples (62%) done (elapsed 51.57 s, remaining 31.08 s) 2023-08-25T02:27:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:31300000 of 50000000 tuples (62%) done (elapsed 51.65 s, remaining 30.86 s) 2023-08-25T02:27:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:31400000 of 50000000 tuples (62%) done (elapsed 51.79 s, remaining 30.68 s) 2023-08-25T02:27:32.921 INFO:teuthology.orchestra.run.smithi089.stderr:31500000 of 50000000 tuples (63%) done (elapsed 52.22 s, remaining 30.67 s) 2023-08-25T02:27:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:31600000 of 50000000 tuples (63%) done (elapsed 52.48 s, remaining 30.56 s) 2023-08-25T02:27:33.305 INFO:teuthology.orchestra.run.smithi089.stderr:31700000 of 50000000 tuples (63%) done (elapsed 52.60 s, remaining 30.37 s) 2023-08-25T02:27:33.387 INFO:teuthology.orchestra.run.smithi089.stderr:31800000 of 50000000 tuples (63%) done (elapsed 52.69 s, remaining 30.15 s) 2023-08-25T02:27:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:31900000 of 50000000 tuples (63%) done (elapsed 52.80 s, remaining 29.96 s) 2023-08-25T02:27:33.604 INFO:teuthology.orchestra.run.smithi089.stderr:32000000 of 50000000 tuples (64%) done (elapsed 52.90 s, remaining 29.76 s) 2023-08-25T02:27:33.717 INFO:teuthology.orchestra.run.smithi089.stderr:32100000 of 50000000 tuples (64%) done (elapsed 53.02 s, remaining 29.56 s) 2023-08-25T02:27:33.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:33 smithi121 ceph-mon[100099]: pgmap v34: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 224 MiB/s wr, 242 op/s 2023-08-25T02:27:33.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:33 smithi122 ceph-mon[100457]: pgmap v34: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 224 MiB/s wr, 242 op/s 2023-08-25T02:27:33.826 INFO:teuthology.orchestra.run.smithi089.stderr:32200000 of 50000000 tuples (64%) done (elapsed 53.12 s, remaining 29.37 s) 2023-08-25T02:27:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:33 smithi089 ceph-mon[98732]: pgmap v34: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 224 MiB/s wr, 242 op/s 2023-08-25T02:27:33.930 INFO:teuthology.orchestra.run.smithi089.stderr:32300000 of 50000000 tuples (64%) done (elapsed 53.23 s, remaining 29.17 s) 2023-08-25T02:27:33.996 INFO:teuthology.orchestra.run.smithi089.stderr:32400000 of 50000000 tuples (64%) done (elapsed 53.29 s, remaining 28.95 s) 2023-08-25T02:27:34.180 INFO:teuthology.orchestra.run.smithi089.stderr:32500000 of 50000000 tuples (65%) done (elapsed 53.48 s, remaining 28.80 s) 2023-08-25T02:27:34.283 INFO:teuthology.orchestra.run.smithi089.stderr:32600000 of 50000000 tuples (65%) done (elapsed 53.58 s, remaining 28.60 s) 2023-08-25T02:27:34.362 INFO:teuthology.orchestra.run.smithi089.stderr:32700000 of 50000000 tuples (65%) done (elapsed 53.66 s, remaining 28.39 s) 2023-08-25T02:27:34.492 INFO:teuthology.orchestra.run.smithi089.stderr:32800000 of 50000000 tuples (65%) done (elapsed 53.79 s, remaining 28.21 s) 2023-08-25T02:27:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:32900000 of 50000000 tuples (65%) done (elapsed 53.85 s, remaining 27.99 s) 2023-08-25T02:27:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:33000000 of 50000000 tuples (66%) done (elapsed 53.96 s, remaining 27.80 s) 2023-08-25T02:27:34.758 INFO:teuthology.orchestra.run.smithi089.stderr:33100000 of 50000000 tuples (66%) done (elapsed 54.06 s, remaining 27.60 s) 2023-08-25T02:27:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:33200000 of 50000000 tuples (66%) done (elapsed 54.17 s, remaining 27.41 s) 2023-08-25T02:27:35.360 INFO:teuthology.orchestra.run.smithi089.stderr:33300000 of 50000000 tuples (66%) done (elapsed 54.66 s, remaining 27.41 s) 2023-08-25T02:27:35.663 INFO:teuthology.orchestra.run.smithi089.stderr:33400000 of 50000000 tuples (66%) done (elapsed 54.96 s, remaining 27.32 s) 2023-08-25T02:27:35.745 INFO:teuthology.orchestra.run.smithi089.stderr:33500000 of 50000000 tuples (67%) done (elapsed 55.04 s, remaining 27.11 s) 2023-08-25T02:27:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:35 smithi089 ceph-mon[98732]: pgmap v35: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 145 MiB/s wr, 205 op/s 2023-08-25T02:27:35.888 INFO:teuthology.orchestra.run.smithi089.stderr:33600000 of 50000000 tuples (67%) done (elapsed 55.19 s, remaining 26.94 s) 2023-08-25T02:27:35.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:35 smithi121 ceph-mon[100099]: pgmap v35: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 145 MiB/s wr, 205 op/s 2023-08-25T02:27:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:35 smithi122 ceph-mon[100457]: pgmap v35: 97 pgs: 97 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 145 MiB/s wr, 205 op/s 2023-08-25T02:27:36.001 INFO:teuthology.orchestra.run.smithi089.stderr:33700000 of 50000000 tuples (67%) done (elapsed 55.30 s, remaining 26.75 s) 2023-08-25T02:27:36.092 INFO:teuthology.orchestra.run.smithi089.stderr:33800000 of 50000000 tuples (67%) done (elapsed 55.39 s, remaining 26.55 s) 2023-08-25T02:27:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:33900000 of 50000000 tuples (67%) done (elapsed 55.50 s, remaining 26.36 s) 2023-08-25T02:27:36.317 INFO:teuthology.orchestra.run.smithi089.stderr:34000000 of 50000000 tuples (68%) done (elapsed 55.62 s, remaining 26.17 s) 2023-08-25T02:27:36.399 INFO:teuthology.orchestra.run.smithi089.stderr:34100000 of 50000000 tuples (68%) done (elapsed 55.70 s, remaining 25.97 s) 2023-08-25T02:27:36.514 INFO:teuthology.orchestra.run.smithi089.stderr:34200000 of 50000000 tuples (68%) done (elapsed 55.81 s, remaining 25.78 s) 2023-08-25T02:27:36.617 INFO:teuthology.orchestra.run.smithi089.stderr:34300000 of 50000000 tuples (68%) done (elapsed 55.91 s, remaining 25.59 s) 2023-08-25T02:27:36.698 INFO:teuthology.orchestra.run.smithi089.stderr:34400000 of 50000000 tuples (68%) done (elapsed 56.00 s, remaining 25.39 s) 2023-08-25T02:27:36.790 INFO:teuthology.orchestra.run.smithi089.stderr:34500000 of 50000000 tuples (69%) done (elapsed 56.09 s, remaining 25.20 s) 2023-08-25T02:27:36.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:27:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:27:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:27:36.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:36 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:27:36.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:27:36.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:27:36.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:27:36.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:36 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:27:36.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:27:36.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:27:36.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:27:36.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:36 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:27:37.385 INFO:teuthology.orchestra.run.smithi089.stderr:34600000 of 50000000 tuples (69%) done (elapsed 56.68 s, remaining 25.23 s) 2023-08-25T02:27:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:34700000 of 50000000 tuples (69%) done (elapsed 56.94 s, remaining 25.11 s) 2023-08-25T02:27:37.751 INFO:teuthology.orchestra.run.smithi089.stderr:34800000 of 50000000 tuples (69%) done (elapsed 57.05 s, remaining 24.92 s) 2023-08-25T02:27:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:37 smithi089 ceph-mon[98732]: pgmap v36: 97 pgs: 97 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 198 MiB/s wr, 276 op/s 2023-08-25T02:27:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:37 smithi121 ceph-mon[100099]: pgmap v36: 97 pgs: 97 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 198 MiB/s wr, 276 op/s 2023-08-25T02:27:37.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:37 smithi122 ceph-mon[100457]: pgmap v36: 97 pgs: 97 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 198 MiB/s wr, 276 op/s 2023-08-25T02:27:38.136 INFO:teuthology.orchestra.run.smithi089.stderr:34900000 of 50000000 tuples (69%) done (elapsed 57.43 s, remaining 24.85 s) 2023-08-25T02:27:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:35000000 of 50000000 tuples (70%) done (elapsed 57.79 s, remaining 24.77 s) 2023-08-25T02:27:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:35100000 of 50000000 tuples (70%) done (elapsed 57.90 s, remaining 24.58 s) 2023-08-25T02:27:38.707 INFO:teuthology.orchestra.run.smithi089.stderr:35200000 of 50000000 tuples (70%) done (elapsed 58.00 s, remaining 24.39 s) 2023-08-25T02:27:38.865 INFO:teuthology.orchestra.run.smithi089.stderr:35300000 of 50000000 tuples (70%) done (elapsed 58.16 s, remaining 24.22 s) 2023-08-25T02:27:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 -- 172.21.15.89:0/2683146642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa99010d170 msgr2=0x7fa99010f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 --2- 172.21.15.89:0/2683146642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa99010d170 0x7fa99010f5a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b0a0 tx=0x7fa98800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 -- 172.21.15.89:0/2683146642 shutdown_connections 2023-08-25T02:27:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 --2- 172.21.15.89:0/2683146642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa990072cf0 0x7fa9900730d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 --2- 172.21.15.89:0/2683146642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa99010d170 0x7fa99010f5a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 --2- 172.21.15.89:0/2683146642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 0x7fa99010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 -- 172.21.15.89:0/2683146642 >> 172.21.15.89:0/2683146642 conn(0x7fa9900fc3e0 msgr2=0x7fa9900fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 -- 172.21.15.89:0/2683146642 shutdown_connections 2023-08-25T02:27:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 -- 172.21.15.89:0/2683146642 wait complete. 2023-08-25T02:27:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.904+0000 7fa9953fe700 1 Processor -- start 2023-08-25T02:27:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 -- start start 2023-08-25T02:27:38.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 0x7fa9901992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 0x7fa99019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99019df60 con 0x7fa990072cf0 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99019e0d0 con 0x7fa99010d170 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa9953fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99019e290 con 0x7fa990073610 2023-08-25T02:27:38.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48554/0 (socket says 172.21.15.89:48554) 2023-08-25T02:27:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98effd700 1 -- 172.21.15.89:0/645765416 learned_addr learned my addr 172.21.15.89:0/645765416 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 0x7fa99019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 0x7fa9901992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 -- 172.21.15.89:0/645765416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 msgr2=0x7fa9901992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 0x7fa9901992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 -- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 msgr2=0x7fa990198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98f7fe700 1 -- 172.21.15.89:0/645765416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa988009d00 con 0x7fa99010d170 2023-08-25T02:27:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.905+0000 7fa98effd700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:38.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 0x7fa99019d920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b070 tx=0x7fa9880078e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa977fff700 1 -- 172.21.15.89:0/645765416 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98800e040 con 0x7fa99010d170 2023-08-25T02:27:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99019e510 con 0x7fa99010d170 2023-08-25T02:27:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa977fff700 1 -- 172.21.15.89:0/645765416 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa98801d470 con 0x7fa99010d170 2023-08-25T02:27:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa977fff700 1 -- 172.21.15.89:0/645765416 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9880052a0 con 0x7fa99010d170 2023-08-25T02:27:38.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.906+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9901a5db0 con 0x7fa99010d170 2023-08-25T02:27:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.908+0000 7fa977fff700 1 -- 172.21.15.89:0/645765416 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 134904+0+0 (secure 0 0 0) 0x7fa988004a50 con 0x7fa99010d170 2023-08-25T02:27:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.909+0000 7fa977fff700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 0x7fa9780a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.909+0000 7fa977fff700 1 -- 172.21.15.89:0/645765416 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa9880ccbd0 con 0x7fa99010d170 2023-08-25T02:27:38.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.909+0000 7fa98effd700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 0x7fa9780a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98effd700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 0x7fa9780a26a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa980011550 tx=0x7fa980009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 Processor -- start 2023-08-25T02:27:38.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 -- start start 2023-08-25T02:27:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e8162320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8162860 0x55c0e815c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e80c4750 con 0x55c0e815c990 2023-08-25T02:27:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e80b5720 con 0x55c0e8162860 2023-08-25T02:27:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e80b3040 con 0x55c0e8161f40 2023-08-25T02:27:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48566/0 (socket says 172.21.15.89:48566) 2023-08-25T02:27:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98f7fe700 1 -- 172.21.15.89:0/3412482853 learned_addr learned my addr 172.21.15.89:0/3412482853 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98effd700 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e8162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.910+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8162860 0x55c0e815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 -- 172.21.15.89:0/3412482853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 msgr2=0x55c0e8162320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e8162320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 -- 172.21.15.89:0/3412482853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8162860 msgr2=0x55c0e815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8162860 0x55c0e815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 -- 172.21.15.89:0/3412482853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0e7f2e5d0 con 0x55c0e815c990 2023-08-25T02:27:38.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b0a0 tx=0x7fa98800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856edc3eccc1f9fa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa9757fa700 1 -- 172.21.15.89:0/3412482853 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98809d070 con 0x55c0e815c990 2023-08-25T02:27:38.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa9757fa700 1 -- 172.21.15.89:0/3412482853 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa988004d00 con 0x55c0e815c990 2023-08-25T02:27:38.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.911+0000 7fa9757fa700 1 -- 172.21.15.89:0/3412482853 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98809df80 con 0x55c0e815c990 2023-08-25T02:27:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/3412482853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 msgr2=0x55c0e815cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b0a0 tx=0x7fa98800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/3412482853 shutdown_connections 2023-08-25T02:27:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e815c990 0x55c0e815cdd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e8162320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/3412482853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8162860 0x55c0e815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/3412482853 >> 172.21.15.89:0/3412482853 conn(0x55c0e80d1080 msgr2=0x55c0e8156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/3412482853 shutdown_connections 2023-08-25T02:27:38.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/3412482853 wait complete. 2023-08-25T02:27:38.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- start start 2023-08-25T02:27:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 Processor -- start 2023-08-25T02:27:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 0x55c0e81686b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 0x55c0e81659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e80ae930 con 0x55c0e8162860 2023-08-25T02:27:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e804cb50 con 0x55c0e8161f40 2023-08-25T02:27:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e804c850 con 0x55c0e815c990 2023-08-25T02:27:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.912+0000 7fa9a15a2b80 10 client.0 _ll_register_callbacks cb 0x55c0e7f96a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98effd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 0x55c0e81686b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 0x55c0e81659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48572/0 (socket says 172.21.15.89:48572) 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98effd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 0x55c0e81686b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40504/0 (socket says 172.21.15.89:40504) 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98f7fe700 1 -- 172.21.15.89:0/2530033404 learned_addr learned my addr 172.21.15.89:0/2530033404 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 -- 172.21.15.89:0/2530033404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 msgr2=0x55c0e81686b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 0x55c0e81686b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 -- 172.21.15.89:0/2530033404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 msgr2=0x55c0e8166040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 -- 172.21.15.89:0/2530033404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9901a5db0 con 0x55c0e8161f40 2023-08-25T02:27:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 0x55c0e81659a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa984012680 tx=0x7fa984012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c5cb42b22ffbc6a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa972ff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:27:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984017070 con 0x55c0e8161f40 2023-08-25T02:27:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c0e812c160 con 0x55c0e8161f40 2023-08-25T02:27:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa984004020 con 0x55c0e8161f40 2023-08-25T02:27:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9840041e0 con 0x55c0e8161f40 2023-08-25T02:27:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa9a15a2b80 10 client.25040 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.913+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c0e7edad80 con 0x55c0e8161f40 2023-08-25T02:27:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c0e812c160 con 0x55c0e8161f40 2023-08-25T02:27:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa98401c590 con 0x55c0e8161f40 2023-08-25T02:27:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 10 client.25040 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 10 client.25040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 20 client.25040 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e8176100 0x55c0e8178520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 4 client.25040 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0e8054b40 con 0x55c0e8176100 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa98effd700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e8176100 0x55c0e8178520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa972ff5700 10 client.25040 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.914+0000 7fa98effd700 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e8176100 0x55c0e8178520 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36fa45e8f61f2c0b server_cookie=1de84018fdefb28e in_seq=0 out_seq=0 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa972ff5700 1 -- 172.21.15.89:0/2530033404 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c0e8054b40 con 0x55c0e8176100 2023-08-25T02:27:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa972ff5700 10 client.25040 handle_command_reply: tid=0 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 client.25040 shutdown 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 msgr2=0x55c0e81659a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 0x55c0e81659a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa984012680 tx=0x7fa984012990 comp rx=0 tx=0).stop 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 shutdown_connections 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e8176100 0x55c0e8178520 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e8166040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa972ff5700 0 client.25040 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e815c990 0x55c0e81686b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2530033404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e8161f40 0x55c0e81659a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 >> 172.21.15.89:0/2530033404 conn(0x55c0e80d1080 msgr2=0x55c0e8151820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 shutdown_connections 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2530033404 wait complete. 2023-08-25T02:27:38.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.915+0000 7fa9a15a2b80 20 client.25040 trim_cache size 0 max 16384 2023-08-25T02:27:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 Processor -- start 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 -- start start 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e816d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e7f308c0 con 0x55c0e8162860 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e8157e80 con 0x55c0e815c990 2023-08-25T02:27:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e816d540 con 0x55c0e8161f40 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa98f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48580/0 (socket says 172.21.15.89:48580) 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa98f7fe700 1 -- 172.21.15.89:0/224153087 learned_addr learned my addr 172.21.15.89:0/224153087 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.930+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e816d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98effd700 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 -- 172.21.15.89:0/224153087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 msgr2=0x55c0e816d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e816d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 -- 172.21.15.89:0/224153087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 msgr2=0x55c0e815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 -- 172.21.15.89:0/224153087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0e7ec3c00 con 0x55c0e8162860 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa988009fd0 tx=0x7fa98800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7ec6089a0024c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa975ffb700 1 -- 172.21.15.89:0/224153087 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98809d070 con 0x55c0e8162860 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa975ffb700 1 -- 172.21.15.89:0/224153087 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa98801d5e0 con 0x55c0e8162860 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.931+0000 7fa975ffb700 1 -- 172.21.15.89:0/224153087 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98809df80 con 0x55c0e8162860 2023-08-25T02:27:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/224153087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 msgr2=0x55c0e815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa988009fd0 tx=0x7fa98800bf40 comp rx=0 tx=0).stop 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/224153087 shutdown_connections 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8162860 0x55c0e815fb30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/224153087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8161f40 0x55c0e816d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/224153087 >> 172.21.15.89:0/224153087 conn(0x55c0e80d1080 msgr2=0x55c0e8171760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/224153087 shutdown_connections 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/224153087 wait complete. 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- start start 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 Processor -- start 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 0x55c0e81650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 0x55c0e81655e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e8165cf0 con 0x55c0e8161f40 2023-08-25T02:27:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e8165e60 con 0x55c0e815c990 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.932+0000 7fa9a15a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0e815d980 con 0x55c0e8162860 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa9a15a2b80 10 client.0 _ll_register_callbacks cb 0x7fa984009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 0x55c0e81650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:38992/0 (socket says 172.21.15.89:38992) 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 -- 172.21.15.89:0/2126039696 learned_addr learned my addr 172.21.15.89:0/2126039696 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 0x55c0e81655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 -- 172.21.15.89:0/2126039696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 msgr2=0x55c0e81655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 0x55c0e81655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 -- 172.21.15.89:0/2126039696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 msgr2=0x55c0e81650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 0x55c0e81650a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 -- 172.21.15.89:0/2126039696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0e7f2e5d0 con 0x55c0e815c990 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98f7fe700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 0x55c0e81655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:27:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 0x55c0e81650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa98effd700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55c0e8178df0 tx=0x7fa9800047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=295b19a09273379d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa972ff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980022040 con 0x55c0e815c990 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c0e814f600 con 0x55c0e815c990 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa98000ea90 con 0x55c0e815c990 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa98000ec90 con 0x55c0e815c990 2023-08-25T02:27:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa9a15a2b80 10 client.25046 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.933+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c0e7f011f0 con 0x55c0e815c990 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c0e814f600 con 0x55c0e815c990 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa9800353f0 con 0x55c0e815c990 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 10 client.25046 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 10 client.25046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 20 client.25046 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e81ca760 0x55c0e81ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 4 client.25046 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c0e80a7910 con 0x55c0e81ca760 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e81ca760 0x55c0e81ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa972ff5700 10 client.25046 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.934+0000 7fa98e7fc700 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e81ca760 0x55c0e81ccb80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fa529d22a26370e server_cookie=d0934743280d48cd in_seq=0 out_seq=0 2023-08-25T02:27:38.957 INFO:teuthology.orchestra.run.smithi089.stderr:35400000 of 50000000 tuples (70%) done (elapsed 58.25 s, remaining 24.03 s) 2023-08-25T02:27:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa972ff5700 1 -- 172.21.15.89:0/2126039696 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c0e80a7910 con 0x55c0e81ca760 2023-08-25T02:27:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa972ff5700 10 client.25046 handle_command_reply: tid=0 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 client.25046 shutdown 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 msgr2=0x55c0e815f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55c0e8178df0 tx=0x7fa9800047c0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 shutdown_connections 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55c0e81ca760 0x55c0e81ccb80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0e815c990 0x55c0e815f130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa972ff5700 0 client.25046 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c0e8162860 0x55c0e81655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 --2- 172.21.15.89:0/2126039696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0e8161f40 0x55c0e81650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 >> 172.21.15.89:0/2126039696 conn(0x55c0e80d1080 msgr2=0x55c0e815fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.962+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 shutdown_connections 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9a15a2b80 1 -- 172.21.15.89:0/2126039696 wait complete. 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9a15a2b80 20 client.25046 trim_cache size 0 max 16384 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:27:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:27:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 msgr2=0x7fa9780a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 0x7fa9780a26a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa980011550 tx=0x7fa980009260 comp rx=0 tx=0).stop 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 msgr2=0x7fa99019d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 0x7fa99019d920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b070 tx=0x7fa9880078e0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 shutdown_connections 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa9780a01e0 0x7fa9780a26a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99010d170 0x7fa99019d920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa990072cf0 0x7fa990198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 --2- 172.21.15.89:0/645765416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa990073610 0x7fa9901992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 >> 172.21.15.89:0/645765416 conn(0x7fa9900fc3e0 msgr2=0x7fa99010f290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 shutdown_connections 2023-08-25T02:27:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:38.963+0000 7fa9953fe700 1 -- 172.21.15.89:0/645765416 wait complete. 2023-08-25T02:27:39.064 INFO:teuthology.orchestra.run.smithi089.stderr:35500000 of 50000000 tuples (71%) done (elapsed 58.36 s, remaining 23.84 s) 2023-08-25T02:27:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:35600000 of 50000000 tuples (71%) done (elapsed 58.48 s, remaining 23.65 s) 2023-08-25T02:27:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:35700000 of 50000000 tuples (71%) done (elapsed 58.57 s, remaining 23.46 s) 2023-08-25T02:27:39.380 INFO:teuthology.orchestra.run.smithi089.stderr:35800000 of 50000000 tuples (71%) done (elapsed 58.68 s, remaining 23.27 s) 2023-08-25T02:27:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:35900000 of 50000000 tuples (71%) done (elapsed 58.78 s, remaining 23.09 s) 2023-08-25T02:27:39.551 INFO:teuthology.orchestra.run.smithi089.stderr:36000000 of 50000000 tuples (72%) done (elapsed 58.85 s, remaining 22.89 s) 2023-08-25T02:27:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:36100000 of 50000000 tuples (72%) done (elapsed 58.97 s, remaining 22.70 s) 2023-08-25T02:27:39.777 INFO:teuthology.orchestra.run.smithi089.stderr:36200000 of 50000000 tuples (72%) done (elapsed 59.07 s, remaining 22.52 s) 2023-08-25T02:27:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:39 smithi089 ceph-mon[98732]: pgmap v37: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 190 MiB/s wr, 286 op/s 2023-08-25T02:27:39.861 INFO:teuthology.orchestra.run.smithi089.stderr:36300000 of 50000000 tuples (72%) done (elapsed 59.16 s, remaining 22.33 s) 2023-08-25T02:27:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:39 smithi121 ceph-mon[100099]: pgmap v37: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 190 MiB/s wr, 286 op/s 2023-08-25T02:27:39.976 INFO:teuthology.orchestra.run.smithi089.stderr:36400000 of 50000000 tuples (72%) done (elapsed 59.27 s, remaining 22.15 s) 2023-08-25T02:27:39.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:39 smithi122 ceph-mon[100457]: pgmap v37: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 190 MiB/s wr, 286 op/s 2023-08-25T02:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:36500000 of 50000000 tuples (73%) done (elapsed 59.37 s, remaining 21.96 s) 2023-08-25T02:27:40.176 INFO:teuthology.orchestra.run.smithi089.stderr:36600000 of 50000000 tuples (73%) done (elapsed 59.47 s, remaining 21.77 s) 2023-08-25T02:27:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:36700000 of 50000000 tuples (73%) done (elapsed 59.59 s, remaining 21.60 s) 2023-08-25T02:27:40.393 INFO:teuthology.orchestra.run.smithi089.stderr:36800000 of 50000000 tuples (73%) done (elapsed 59.69 s, remaining 21.41 s) 2023-08-25T02:27:40.496 INFO:teuthology.orchestra.run.smithi089.stderr:36900000 of 50000000 tuples (73%) done (elapsed 59.79 s, remaining 21.23 s) 2023-08-25T02:27:40.611 INFO:teuthology.orchestra.run.smithi089.stderr:37000000 of 50000000 tuples (74%) done (elapsed 59.91 s, remaining 21.05 s) 2023-08-25T02:27:40.705 INFO:teuthology.orchestra.run.smithi089.stderr:37100000 of 50000000 tuples (74%) done (elapsed 60.00 s, remaining 20.86 s) 2023-08-25T02:27:41.247 INFO:teuthology.orchestra.run.smithi089.stderr:37200000 of 50000000 tuples (74%) done (elapsed 60.54 s, remaining 20.83 s) 2023-08-25T02:27:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:37300000 of 50000000 tuples (74%) done (elapsed 60.85 s, remaining 20.72 s) 2023-08-25T02:27:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:41 smithi089 ceph-mon[98732]: pgmap v38: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 148 MiB/s wr, 176 op/s 2023-08-25T02:27:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:37400000 of 50000000 tuples (74%) done (elapsed 60.92 s, remaining 20.52 s) 2023-08-25T02:27:41.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:41 smithi121 ceph-mon[100099]: pgmap v38: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 148 MiB/s wr, 176 op/s 2023-08-25T02:27:41.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:41 smithi122 ceph-mon[100457]: pgmap v38: 97 pgs: 97 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 148 MiB/s wr, 176 op/s 2023-08-25T02:27:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:37500000 of 50000000 tuples (75%) done (elapsed 61.06 s, remaining 20.35 s) 2023-08-25T02:27:41.865 INFO:teuthology.orchestra.run.smithi089.stderr:37600000 of 50000000 tuples (75%) done (elapsed 61.16 s, remaining 20.17 s) 2023-08-25T02:27:41.931 INFO:teuthology.orchestra.run.smithi089.stderr:37700000 of 50000000 tuples (75%) done (elapsed 61.23 s, remaining 19.98 s) 2023-08-25T02:27:42.039 INFO:teuthology.orchestra.run.smithi089.stderr:37800000 of 50000000 tuples (75%) done (elapsed 61.34 s, remaining 19.80 s) 2023-08-25T02:27:42.134 INFO:teuthology.orchestra.run.smithi089.stderr:37900000 of 50000000 tuples (75%) done (elapsed 61.43 s, remaining 19.61 s) 2023-08-25T02:27:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:38000000 of 50000000 tuples (76%) done (elapsed 61.53 s, remaining 19.43 s) 2023-08-25T02:27:42.340 INFO:teuthology.orchestra.run.smithi089.stderr:38100000 of 50000000 tuples (76%) done (elapsed 61.64 s, remaining 19.25 s) 2023-08-25T02:27:42.442 INFO:teuthology.orchestra.run.smithi089.stderr:38200000 of 50000000 tuples (76%) done (elapsed 61.74 s, remaining 19.07 s) 2023-08-25T02:27:42.527 INFO:teuthology.orchestra.run.smithi089.stderr:38300000 of 50000000 tuples (76%) done (elapsed 61.82 s, remaining 18.89 s) 2023-08-25T02:27:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:38400000 of 50000000 tuples (76%) done (elapsed 61.92 s, remaining 18.71 s) 2023-08-25T02:27:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:38500000 of 50000000 tuples (77%) done (elapsed 62.03 s, remaining 18.53 s) 2023-08-25T02:27:42.857 INFO:teuthology.orchestra.run.smithi089.stderr:38600000 of 50000000 tuples (77%) done (elapsed 62.15 s, remaining 18.36 s) 2023-08-25T02:27:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:38700000 of 50000000 tuples (77%) done (elapsed 62.25 s, remaining 18.18 s) 2023-08-25T02:27:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:38800000 of 50000000 tuples (77%) done (elapsed 62.34 s, remaining 17.99 s) 2023-08-25T02:27:43.152 INFO:teuthology.orchestra.run.smithi089.stderr:38900000 of 50000000 tuples (77%) done (elapsed 62.45 s, remaining 17.82 s) 2023-08-25T02:27:43.250 INFO:teuthology.orchestra.run.smithi089.stderr:39000000 of 50000000 tuples (78%) done (elapsed 62.55 s, remaining 17.64 s) 2023-08-25T02:27:43.553 INFO:teuthology.orchestra.run.smithi089.stderr:39100000 of 50000000 tuples (78%) done (elapsed 62.85 s, remaining 17.52 s) 2023-08-25T02:27:43.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:43 smithi121 ceph-mon[100099]: pgmap v39: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 MiB/s wr, 279 op/s 2023-08-25T02:27:43.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:43 smithi122 ceph-mon[100457]: pgmap v39: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 MiB/s wr, 279 op/s 2023-08-25T02:27:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:43 smithi089 ceph-mon[98732]: pgmap v39: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 MiB/s wr, 279 op/s 2023-08-25T02:27:45.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:45 smithi121 ceph-mon[100099]: pgmap v40: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 MiB/s wr, 199 op/s 2023-08-25T02:27:45.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:45 smithi122 ceph-mon[100457]: pgmap v40: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 MiB/s wr, 199 op/s 2023-08-25T02:27:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:45 smithi089 ceph-mon[98732]: pgmap v40: 97 pgs: 97 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 MiB/s wr, 199 op/s 2023-08-25T02:27:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:47 smithi121 ceph-mon[100099]: pgmap v41: 97 pgs: 97 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 153 MiB/s wr, 255 op/s 2023-08-25T02:27:47.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:47 smithi122 ceph-mon[100457]: pgmap v41: 97 pgs: 97 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 153 MiB/s wr, 255 op/s 2023-08-25T02:27:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:39200000 of 50000000 tuples (78%) done (elapsed 67.10 s, remaining 18.49 s) 2023-08-25T02:27:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:47 smithi089 ceph-mon[98732]: pgmap v41: 97 pgs: 97 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 153 MiB/s wr, 255 op/s 2023-08-25T02:27:47.873 INFO:teuthology.orchestra.run.smithi089.stderr:39300000 of 50000000 tuples (78%) done (elapsed 67.17 s, remaining 18.29 s) 2023-08-25T02:27:47.990 INFO:teuthology.orchestra.run.smithi089.stderr:39400000 of 50000000 tuples (78%) done (elapsed 67.29 s, remaining 18.10 s) 2023-08-25T02:27:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:39500000 of 50000000 tuples (79%) done (elapsed 67.38 s, remaining 17.91 s) 2023-08-25T02:27:48.419 INFO:teuthology.orchestra.run.smithi089.stderr:39600000 of 50000000 tuples (79%) done (elapsed 67.72 s, remaining 17.78 s) 2023-08-25T02:27:49.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:49 smithi121 ceph-mon[100099]: pgmap v42: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 108 MiB/s wr, 187 op/s 2023-08-25T02:27:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:49 smithi122 ceph-mon[100457]: pgmap v42: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 108 MiB/s wr, 187 op/s 2023-08-25T02:27:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:49 smithi089 ceph-mon[98732]: pgmap v42: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 108 MiB/s wr, 187 op/s 2023-08-25T02:27:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:39700000 of 50000000 tuples (79%) done (elapsed 70.45 s, remaining 18.28 s) 2023-08-25T02:27:51.255 INFO:teuthology.orchestra.run.smithi089.stderr:39800000 of 50000000 tuples (79%) done (elapsed 70.55 s, remaining 18.08 s) 2023-08-25T02:27:51.360 INFO:teuthology.orchestra.run.smithi089.stderr:39900000 of 50000000 tuples (79%) done (elapsed 70.66 s, remaining 17.89 s) 2023-08-25T02:27:51.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:51 smithi121 ceph-mon[100099]: pgmap v43: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 88 MiB/s wr, 160 op/s 2023-08-25T02:27:51.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:51 smithi122 ceph-mon[100457]: pgmap v43: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 88 MiB/s wr, 160 op/s 2023-08-25T02:27:51.504 INFO:teuthology.orchestra.run.smithi089.stderr:40000000 of 50000000 tuples (80%) done (elapsed 70.80 s, remaining 17.70 s) 2023-08-25T02:27:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:51 smithi089 ceph-mon[98732]: pgmap v43: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 88 MiB/s wr, 160 op/s 2023-08-25T02:27:51.610 INFO:teuthology.orchestra.run.smithi089.stderr:40100000 of 50000000 tuples (80%) done (elapsed 70.91 s, remaining 17.51 s) 2023-08-25T02:27:52.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 -- 172.21.15.89:0/2917694129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 msgr2=0x7f2c1c0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2917694129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c0fff00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2c180099c0 tx=0x7f2c18009cd0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 -- 172.21.15.89:0/2917694129 shutdown_connections 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2917694129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c105840 0x7f2c1c105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2917694129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c100440 0x7f2c1c10c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2917694129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c0fff00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 -- 172.21.15.89:0/2917694129 >> 172.21.15.89:0/2917694129 conn(0x7f2c1c075250 msgr2=0x7f2c1c075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 -- 172.21.15.89:0/2917694129 shutdown_connections 2023-08-25T02:27:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 -- 172.21.15.89:0/2917694129 wait complete. 2023-08-25T02:27:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.072+0000 7f2c2422e700 1 Processor -- start 2023-08-25T02:27:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 -- start start 2023-08-25T02:27:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c100440 0x7f2c1c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c105840 0x7f2c1c195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c195cd0 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c195e40 con 0x7f2c1c100440 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c2422e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c196000 con 0x7f2c1c105840 2023-08-25T02:27:52.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48596/0 (socket says 172.21.15.89:48596) 2023-08-25T02:27:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 -- 172.21.15.89:0/2309665937 learned_addr learned my addr 172.21.15.89:0/2309665937 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c217c9700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c100440 0x7f2c1c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c227cb700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c105840 0x7f2c1c195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 -- 172.21.15.89:0/2309665937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c105840 msgr2=0x7f2c1c195660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c105840 0x7f2c1c195660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 -- 172.21.15.89:0/2309665937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c100440 msgr2=0x7f2c1c19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c100440 0x7f2c1c19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 -- 172.21.15.89:0/2309665937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180095f0 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.073+0000 7f2c21fca700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9a0 tx=0x7f2c0c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.074+0000 7f2c12ffd700 1 -- 172.21.15.89:0/2309665937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c009790 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.074+0000 7f2c12ffd700 1 -- 172.21.15.89:0/2309665937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c0c010430 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.074+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c1962e0 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.074+0000 7f2c12ffd700 1 -- 172.21.15.89:0/2309665937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c018680 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.074+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c1a27f0 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.076+0000 7f2c12ffd700 1 -- 172.21.15.89:0/2309665937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 134904+0+0 (secure 0 0 0) 0x7f2c0c0105a0 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.077+0000 7f2c12ffd700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 0x7f2c080a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.077+0000 7f2c217c9700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 0x7f2c080a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.077+0000 7f2c12ffd700 1 -- 172.21.15.89:0/2309665937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2c0c014070 con 0x7f2c1c0ffa80 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.077+0000 7f2c217c9700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 0x7f2c080a26a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009990 tx=0x7f2c18005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 Processor -- start 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 -- start start 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1dc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1de260 0x55de2a1dc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a144750 con 0x55de2a1dcf70 2023-08-25T02:27:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a135720 con 0x55de2a1de260 2023-08-25T02:27:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a133040 con 0x55de2a1dde80 2023-08-25T02:27:52.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48608/0 (socket says 172.21.15.89:48608) 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 -- 172.21.15.89:0/3515102825 learned_addr learned my addr 172.21.15.89:0/3515102825 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c21fca700 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c217c9700 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1de260 0x55de2a1dc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 -- 172.21.15.89:0/3515102825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 msgr2=0x55de2a1dc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1dc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 -- 172.21.15.89:0/3515102825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1de260 msgr2=0x55de2a1dc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1de260 0x55de2a1dc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.078+0000 7f2c227cb700 1 -- 172.21.15.89:0/3515102825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de29fae5d0 con 0x55de2a1dcf70 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c227cb700 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2c14011600 tx=0x7f2c14009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafbf8947d0dac02 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c07fff700 1 -- 172.21.15.89:0/3515102825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400e040 con 0x55de2a1dcf70 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c07fff700 1 -- 172.21.15.89:0/3515102825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c1400f040 con 0x55de2a1dcf70 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c07fff700 1 -- 172.21.15.89:0/3515102825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14004690 con 0x55de2a1dcf70 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 -- 172.21.15.89:0/3515102825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 msgr2=0x55de2a1ecae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2c14011600 tx=0x7f2c14009920 comp rx=0 tx=0).stop 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 -- 172.21.15.89:0/3515102825 shutdown_connections 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1ecae0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1dc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/3515102825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1de260 0x55de2a1dc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.079+0000 7f2c303d2b80 1 -- 172.21.15.89:0/3515102825 >> 172.21.15.89:0/3515102825 conn(0x55de2a151080 msgr2=0x55de2a1d6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 -- 172.21.15.89:0/3515102825 shutdown_connections 2023-08-25T02:27:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 -- 172.21.15.89:0/3515102825 wait complete. 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 -- start start 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 Processor -- start 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.080+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1ea390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1de260 0x55de2a1ea8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a12e930 con 0x55de2a1dcf70 2023-08-25T02:27:52.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a0ccb50 con 0x55de2a1dde80 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a0cc850 con 0x55de2a1de260 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 10 client.0 _ll_register_callbacks cb 0x55de2a016a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48610/0 (socket says 172.21.15.89:48610) 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 -- 172.21.15.89:0/1310770790 learned_addr learned my addr 172.21.15.89:0/1310770790 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c227cb700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1de260 0x55de2a1ea8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c217c9700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 -- 172.21.15.89:0/1310770790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1de260 msgr2=0x55de2a1ea8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1de260 0x55de2a1ea8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 -- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 msgr2=0x55de2a1ea390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1ea390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 -- 172.21.15.89:0/1310770790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c140112b0 con 0x55de2a1dcf70 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c21fca700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c098f10 tx=0x7f2c0c0994c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508f8f2abfc3bcaa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c057fa700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0a0b00 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2c0c09cdb0 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de2a1ac160 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c0c0a0d90 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 10 client.15429 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.081+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de29f5ad80 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de2a1ac160 con 0x55de2a1dcf70 2023-08-25T02:27:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f2c0c093d20 con 0x55de2a1dcf70 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 10 client.15429 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 10 client.15429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 20 client.15429 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a1fa620 0x55de2a1fca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 4 client.15429 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de2a0d4b40 con 0x55de2a1fa620 2023-08-25T02:27:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c217c9700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a1fa620 0x55de2a1fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c057fa700 10 client.15429 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.082+0000 7f2c217c9700 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a1fa620 0x55de2a1fca40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b91ac99236fe5cf server_cookie=d2cd8db01eef6d98 in_seq=0 out_seq=0 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c057fa700 1 -- 172.21.15.89:0/1310770790 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55de2a0d4b40 con 0x55de2a1fa620 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c057fa700 10 client.15429 handle_command_reply: tid=0 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 client.15429 shutdown 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 msgr2=0x55de2a1e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c098f10 tx=0x7f2c0c0994c0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 shutdown_connections 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a1fa620 0x55de2a1fca40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1de260 0x55de2a1ea8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1dcf70 0x55de2a1e9e50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/1310770790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c057fa700 0 client.15429 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 >> 172.21.15.89:0/1310770790 conn(0x55de2a151080 msgr2=0x55de2a1d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 shutdown_connections 2023-08-25T02:27:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 1 -- 172.21.15.89:0/1310770790 wait complete. 2023-08-25T02:27:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.083+0000 7f2c303d2b80 20 client.15429 trim_cache size 0 max 16384 2023-08-25T02:27:52.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 Processor -- start 2023-08-25T02:27:52.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 -- start start 2023-08-25T02:27:52.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dcf70 0x55de2a1f1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29fb08c0 con 0x55de2a1de260 2023-08-25T02:27:52.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a1d7280 con 0x55de2a1dcf70 2023-08-25T02:27:52.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a1f1be0 con 0x55de2a1dde80 2023-08-25T02:27:52.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c217c9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48622/0 (socket says 172.21.15.89:48622) 2023-08-25T02:27:52.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 -- 172.21.15.89:0/732709042 learned_addr learned my addr 172.21.15.89:0/732709042 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:52.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c21fca700 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dcf70 0x55de2a1f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 -- 172.21.15.89:0/732709042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 msgr2=0x55de2a1ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 -- 172.21.15.89:0/732709042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dcf70 msgr2=0x55de2a1f1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dcf70 0x55de2a1f1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 -- 172.21.15.89:0/732709042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de29f43c00 con 0x55de2a1de260 2023-08-25T02:27:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.098+0000 7f2c227cb700 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2c14011600 tx=0x7f2c14006020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d342785614ce340c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c10ff9700 1 -- 172.21.15.89:0/732709042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400e050 con 0x55de2a1de260 2023-08-25T02:27:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c10ff9700 1 -- 172.21.15.89:0/732709042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c140092e0 con 0x55de2a1de260 2023-08-25T02:27:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c10ff9700 1 -- 172.21.15.89:0/732709042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140057f0 con 0x55de2a1de260 2023-08-25T02:27:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- 172.21.15.89:0/732709042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 msgr2=0x55de2a1dc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2c14011600 tx=0x7f2c14006020 comp rx=0 tx=0).stop 2023-08-25T02:27:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- 172.21.15.89:0/732709042 shutdown_connections 2023-08-25T02:27:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dcf70 0x55de2a1f1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1dc450 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/732709042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dde80 0x55de2a1ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- 172.21.15.89:0/732709042 >> 172.21.15.89:0/732709042 conn(0x55de2a151080 msgr2=0x55de2a1e5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- 172.21.15.89:0/732709042 shutdown_connections 2023-08-25T02:27:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- 172.21.15.89:0/732709042 wait complete. 2023-08-25T02:27:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.099+0000 7f2c303d2b80 1 -- start start 2023-08-25T02:27:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 Processor -- start 2023-08-25T02:27:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dcf70 0x55de2a1f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a1edc70 con 0x55de2a1de260 2023-08-25T02:27:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a1edde0 con 0x55de2a1dde80 2023-08-25T02:27:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2a1edfa0 con 0x55de2a1dcf70 2023-08-25T02:27:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c303d2b80 10 client.0 _ll_register_callbacks cb 0x7f2c0c098b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:27:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c21fca700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dcf70 0x55de2a1f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48628/0 (socket says 172.21.15.89:48628) 2023-08-25T02:27:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 -- 172.21.15.89:0/953408844 learned_addr learned my addr 172.21.15.89:0/953408844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:27:52.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c217c9700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 -- 172.21.15.89:0/953408844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dcf70 msgr2=0x55de2a1f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dcf70 0x55de2a1f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 -- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 msgr2=0x55de2a1f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:27:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.100+0000 7f2c227cb700 1 -- 172.21.15.89:0/953408844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de29fae5d0 con 0x55de2a1de260 2023-08-25T02:27:52.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c227cb700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009b00 tx=0x7f2c14005b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacf5f9a5de12318 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:27:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:27:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1402a020 con 0x55de2a1de260 2023-08-25T02:27:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2c14003e10 con 0x55de2a1de260 2023-08-25T02:27:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de2a1cf600 con 0x55de2a1de260 2023-08-25T02:27:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c1400e020 con 0x55de2a1de260 2023-08-25T02:27:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 10 client.15435 fetch_fsmap learned FSMap version 28 2023-08-25T02:27:52.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de29f811f0 con 0x55de2a1de260 2023-08-25T02:27:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de2a1cf600 con 0x55de2a1de260 2023-08-25T02:27:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f2c140154e0 con 0x55de2a1de260 2023-08-25T02:27:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 10 client.15435 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:27:52.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 10 client.15435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:27:52.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 20 client.15435 populate_metadata read hostname 'smithi089' 2023-08-25T02:27:52.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a24f1c0 0x55de2a2515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:27:52.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 4 client.15435 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:27:52.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55de2a127910 con 0x55de2a24f1c0 2023-08-25T02:27:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.101+0000 7f2c21fca700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a24f1c0 0x55de2a2515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:27:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.102+0000 7f2c057fa700 10 client.15435 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.102+0000 7f2c21fca700 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a24f1c0 0x55de2a2515e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c5e4e4cd5668cb server_cookie=d71e63393b928845 in_seq=0 out_seq=0 2023-08-25T02:27:52.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c057fa700 1 -- 172.21.15.89:0/953408844 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55de2a127910 con 0x55de2a24f1c0 2023-08-25T02:27:52.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c057fa700 10 client.15435 handle_command_reply: tid=0 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 client.15435 shutdown 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 msgr2=0x55de2a1ed650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009b00 tx=0x7f2c14005b60 comp rx=0 tx=0).stop 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 shutdown_connections 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55de2a24f1c0 0x55de2a2515e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de2a1de260 0x55de2a1ed650 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55de2a1dcf70 0x55de2a1f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 --2- 172.21.15.89:0/953408844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de2a1dde80 0x55de2a1f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c057fa700 0 client.15435 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 >> 172.21.15.89:0/953408844 conn(0x55de2a151080 msgr2=0x55de2a1d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 shutdown_connections 2023-08-25T02:27:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 1 -- 172.21.15.89:0/953408844 wait complete. 2023-08-25T02:27:52.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.137+0000 7f2c303d2b80 20 client.15435 trim_cache size 0 max 16384 2023-08-25T02:27:52.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:27:52.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:27:52.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:27:52.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:27:52.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 msgr2=0x7f2c080a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 0x7f2c080a26a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009990 tx=0x7f2c18005fb0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 msgr2=0x7f2c1c19b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9a0 tx=0x7f2c0c00ecb0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 shutdown_connections 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2c080a01e0 0x7f2c080a26a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c1c105840 0x7f2c1c195660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c1c0ffa80 0x7f2c1c19b800 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 --2- 172.21.15.89:0/2309665937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c100440 0x7f2c1c19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:27:52.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 >> 172.21.15.89:0/2309665937 conn(0x7f2c1c075250 msgr2=0x7f2c1c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:27:52.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 shutdown_connections 2023-08-25T02:27:52.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:27:52.138+0000 7f2c2422e700 1 -- 172.21.15.89:0/2309665937 wait complete. 2023-08-25T02:27:52.423 INFO:teuthology.orchestra.run.smithi089.stderr:40200000 of 50000000 tuples (80%) done (elapsed 71.72 s, remaining 17.48 s) 2023-08-25T02:27:52.525 INFO:teuthology.orchestra.run.smithi089.stderr:40300000 of 50000000 tuples (80%) done (elapsed 71.82 s, remaining 17.29 s) 2023-08-25T02:27:52.618 INFO:teuthology.orchestra.run.smithi089.stderr:40400000 of 50000000 tuples (80%) done (elapsed 71.92 s, remaining 17.09 s) 2023-08-25T02:27:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:40500000 of 50000000 tuples (81%) done (elapsed 72.34 s, remaining 16.97 s) 2023-08-25T02:27:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:53 smithi121 ceph-mon[100099]: pgmap v44: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 96 MiB/s wr, 173 op/s 2023-08-25T02:27:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:53 smithi122 ceph-mon[100457]: pgmap v44: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 96 MiB/s wr, 173 op/s 2023-08-25T02:27:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:53 smithi089 ceph-mon[98732]: pgmap v44: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 96 MiB/s wr, 173 op/s 2023-08-25T02:27:54.436 INFO:teuthology.orchestra.run.smithi089.stderr:40600000 of 50000000 tuples (81%) done (elapsed 73.73 s, remaining 17.07 s) 2023-08-25T02:27:54.512 INFO:teuthology.orchestra.run.smithi089.stderr:40700000 of 50000000 tuples (81%) done (elapsed 73.81 s, remaining 16.87 s) 2023-08-25T02:27:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:55 smithi121 ceph-mon[100099]: pgmap v45: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 36 MiB/s wr, 71 op/s 2023-08-25T02:27:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:55 smithi122 ceph-mon[100457]: pgmap v45: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 36 MiB/s wr, 71 op/s 2023-08-25T02:27:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:55 smithi089 ceph-mon[98732]: pgmap v45: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 36 MiB/s wr, 71 op/s 2023-08-25T02:27:57.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:57 smithi121 ceph-mon[100099]: pgmap v46: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 43 MiB/s wr, 123 op/s 2023-08-25T02:27:57.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:57 smithi122 ceph-mon[100457]: pgmap v46: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 43 MiB/s wr, 123 op/s 2023-08-25T02:27:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:57 smithi089 ceph-mon[98732]: pgmap v46: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 43 MiB/s wr, 123 op/s 2023-08-25T02:27:58.522 INFO:teuthology.orchestra.run.smithi089.stderr:40800000 of 50000000 tuples (81%) done (elapsed 77.82 s, remaining 17.55 s) 2023-08-25T02:27:58.641 INFO:teuthology.orchestra.run.smithi089.stderr:40900000 of 50000000 tuples (81%) done (elapsed 77.94 s, remaining 17.34 s) 2023-08-25T02:27:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:27:59 smithi121 ceph-mon[100099]: pgmap v47: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 24 MiB/s wr, 80 op/s 2023-08-25T02:27:59.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:27:59 smithi122 ceph-mon[100457]: pgmap v47: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 24 MiB/s wr, 80 op/s 2023-08-25T02:27:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:27:59 smithi089 ceph-mon[98732]: pgmap v47: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 24 MiB/s wr, 80 op/s 2023-08-25T02:28:01.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:01 smithi121 ceph-mon[100099]: pgmap v48: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 16 MiB/s wr, 78 op/s 2023-08-25T02:28:01.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:01 smithi122 ceph-mon[100457]: pgmap v48: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 16 MiB/s wr, 78 op/s 2023-08-25T02:28:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:01 smithi089 ceph-mon[98732]: pgmap v48: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 16 MiB/s wr, 78 op/s 2023-08-25T02:28:02.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:02 smithi121 ceph-mon[100099]: mgrmap e23: x(active, since 90s), standbys: y, z 2023-08-25T02:28:02.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:02 smithi122 ceph-mon[100457]: mgrmap e23: x(active, since 90s), standbys: y, z 2023-08-25T02:28:02.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:02 smithi089 ceph-mon[98732]: mgrmap e23: x(active, since 90s), standbys: y, z 2023-08-25T02:28:03.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:03 smithi121 ceph-mon[100099]: pgmap v49: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 17 MiB/s wr, 97 op/s 2023-08-25T02:28:03.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:03 smithi122 ceph-mon[100457]: pgmap v49: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 17 MiB/s wr, 97 op/s 2023-08-25T02:28:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:03 smithi089 ceph-mon[98732]: pgmap v49: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 17 MiB/s wr, 97 op/s 2023-08-25T02:28:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.348+0000 7fe6a05d9700 1 -- 172.21.15.89:0/41249001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 msgr2=0x7fe69810c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.348+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/41249001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69810c6c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe69000b0a0 tx=0x7fe69000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 -- 172.21.15.89:0/41249001 shutdown_connections 2023-08-25T02:28:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/41249001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe698105800 0x7fe698105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/41249001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69810c6c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/41249001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6980ff9b0 0x7fe6980ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 -- 172.21.15.89:0/41249001 >> 172.21.15.89:0/41249001 conn(0x7fe698077750 msgr2=0x7fe698077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 -- 172.21.15.89:0/41249001 shutdown_connections 2023-08-25T02:28:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.349+0000 7fe6a05d9700 1 -- 172.21.15.89:0/41249001 wait complete. 2023-08-25T02:28:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 Processor -- start 2023-08-25T02:28:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 -- start start 2023-08-25T02:28:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6980ff9b0 0x7fe69819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe698105800 0x7fe698196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe698196f80 con 0x7fe698100370 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6981970f0 con 0x7fe6980ff9b0 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe6a05d9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6981972b0 con 0x7fe698105800 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe69db74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe69db74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48640/0 (socket says 172.21.15.89:48640) 2023-08-25T02:28:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.350+0000 7fe69db74700 1 -- 172.21.15.89:0/241277827 learned_addr learned my addr 172.21.15.89:0/241277827 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 -- 172.21.15.89:0/241277827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe698105800 msgr2=0x7fe698196910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-08-25T02:28:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe698105800 0x7fe698196910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 -- 172.21.15.89:0/241277827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6980ff9b0 msgr2=0x7fe69819cab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6980ff9b0 0x7fe69819cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 -- 172.21.15.89:0/241277827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe690009d00 con 0x7fe698100370 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe69db74700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe68800b530 tx=0x7fe68800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe68f7fe700 1 -- 172.21.15.89:0/241277827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688011830 con 0x7fe698100370 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe698197590 con 0x7fe698100370 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe68f7fe700 1 -- 172.21.15.89:0/241277827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe688011e50 con 0x7fe698100370 2023-08-25T02:28:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe68f7fe700 1 -- 172.21.15.89:0/241277827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68800f2e0 con 0x7fe698100370 2023-08-25T02:28:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.351+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6981a3aa0 con 0x7fe698100370 2023-08-25T02:28:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.355+0000 7fe68f7fe700 1 -- 172.21.15.89:0/241277827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fe688010580 con 0x7fe698100370 2023-08-25T02:28:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.356+0000 7fe68f7fe700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 0x7fe6840a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.356+0000 7fe69e375700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 0x7fe6840a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.356+0000 7fe68f7fe700 1 -- 172.21.15.89:0/241277827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe6880caff0 con 0x7fe698100370 2023-08-25T02:28:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.356+0000 7fe6ac77db80 1 Processor -- start 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 -- start start 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe69e375700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 0x7fe6840a2600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe694009730 tx=0x7fe694006cb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea6470300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea646c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea63d4750 con 0x559ea646ce10 2023-08-25T02:28:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea63c5720 con 0x559ea646c450 2023-08-25T02:28:05.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea63c3040 con 0x559ea646ff20 2023-08-25T02:28:05.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.357+0000 7fe69eb76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48650/0 (socket says 172.21.15.89:48650) 2023-08-25T02:28:05.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 -- 172.21.15.89:0/3675349060 learned_addr learned my addr 172.21.15.89:0/3675349060 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69e375700 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea6470300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 -- 172.21.15.89:0/3675349060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 msgr2=0x559ea6470300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea6470300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 -- 172.21.15.89:0/3675349060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 msgr2=0x559ea646c8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea646c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 -- 172.21.15.89:0/3675349060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ea623e5d0 con 0x559ea646ce10 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69e375700 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea6470300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.358+0000 7fe69eb76700 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe690009fd0 tx=0x7fe690005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f404bd86f6824fa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.359+0000 7fe68cff9700 1 -- 172.21.15.89:0/3675349060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69000e040 con 0x559ea646ce10 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.359+0000 7fe68cff9700 1 -- 172.21.15.89:0/3675349060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6900093d0 con 0x559ea646ce10 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.359+0000 7fe68cff9700 1 -- 172.21.15.89:0/3675349060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6900052b0 con 0x559ea646ce10 2023-08-25T02:28:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3675349060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 msgr2=0x559ea64810c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe690009fd0 tx=0x7fe690005ac0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3675349060 shutdown_connections 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea64810c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea6470300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3675349060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea646c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3675349060 >> 172.21.15.89:0/3675349060 conn(0x559ea63e1080 msgr2=0x559ea6466d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3675349060 shutdown_connections 2023-08-25T02:28:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.360+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3675349060 wait complete. 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.361+0000 7fe6ac77db80 1 -- start start 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.361+0000 7fe6ac77db80 1 Processor -- start 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea647b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea63be930 con 0x559ea646ff20 2023-08-25T02:28:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea635cb50 con 0x559ea646c450 2023-08-25T02:28:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea635c850 con 0x559ea646ce10 2023-08-25T02:28:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe6ac77db80 10 client.0 _ll_register_callbacks cb 0x559ea62a6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe69eb76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe69eb76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48656/0 (socket says 172.21.15.89:48656) 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.362+0000 7fe69eb76700 1 -- 172.21.15.89:0/2896641836 learned_addr learned my addr 172.21.15.89:0/2896641836 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69e375700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 -- 172.21.15.89:0/2896641836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 msgr2=0x559ea647b360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea647b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 -- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 msgr2=0x559ea647ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 -- 172.21.15.89:0/2896641836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69810cfc0 con 0x559ea646ff20 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe69eb76700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe690007a50 tx=0x7fe6900077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45e28af520d5e07 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe681ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69000e040 con 0x559ea646ff20 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe690005070 con 0x559ea646ff20 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ea643c160 con 0x559ea646ff20 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe690005230 con 0x559ea646ff20 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe6ac77db80 10 client.15447 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.363+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559ea61ead80 con 0x559ea646ff20 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559ea643c160 con 0x559ea646ff20 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fe690019070 con 0x559ea646ff20 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 10 client.15447 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 10 client.15447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 20 client.15447 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64a7f40 0x559ea64aa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 4 client.15447 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:28:05.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.364+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ea6364b40 con 0x559ea64a7f40 2023-08-25T02:28:05.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.365+0000 7fe69e375700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64a7f40 0x559ea64aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.366+0000 7fe69e375700 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64a7f40 0x559ea64aa330 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5e1a0bfa8b700f server_cookie=236c5fc4b255e3ef in_seq=0 out_seq=0 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.366+0000 7fe681ffb700 10 client.15447 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.367+0000 7fe681ffb700 1 -- 172.21.15.89:0/2896641836 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559ea6364b40 con 0x559ea64a7f40 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.367+0000 7fe681ffb700 10 client.15447 handle_command_reply: tid=0 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.367+0000 7fe6ac77db80 1 client.15447 shutdown 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.367+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 msgr2=0x559ea6480b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.367+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe690007a50 tx=0x7fe6900077f0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 shutdown_connections 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64a7f40 0x559ea64aa330 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ff20 0x559ea6480b40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2896641836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea647b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe681ffb700 0 client.15447 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 >> 172.21.15.89:0/2896641836 conn(0x559ea63e1080 msgr2=0x559ea6461820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 shutdown_connections 2023-08-25T02:28:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2896641836 wait complete. 2023-08-25T02:28:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.368+0000 7fe6ac77db80 20 client.15447 trim_cache size 0 max 16384 2023-08-25T02:28:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.387+0000 7fe6ac77db80 1 Processor -- start 2023-08-25T02:28:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.387+0000 7fe6ac77db80 1 -- start start 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea6476060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646ff20 0x559ea647a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea62408c0 con 0x559ea646c450 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea6477780 con 0x559ea646ff20 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea6470f20 con 0x559ea646ce10 2023-08-25T02:28:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48660/0 (socket says 172.21.15.89:48660) 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 -- 172.21.15.89:0/2664513239 learned_addr learned my addr 172.21.15.89:0/2664513239 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 -- 172.21.15.89:0/2664513239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 msgr2=0x559ea6476060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea6476060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 -- 172.21.15.89:0/2664513239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646ff20 msgr2=0x559ea647a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646ff20 0x559ea647a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 -- 172.21.15.89:0/2664513239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ea6470630 con 0x559ea646c450 2023-08-25T02:28:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe69e375700 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe694013710 tx=0x7fe694011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e182c4991db5c4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.388+0000 7fe68d7fa700 1 -- 172.21.15.89:0/2664513239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694013920 con 0x559ea646c450 2023-08-25T02:28:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe68d7fa700 1 -- 172.21.15.89:0/2664513239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe694023040 con 0x559ea646c450 2023-08-25T02:28:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe68d7fa700 1 -- 172.21.15.89:0/2664513239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69402c9b0 con 0x559ea646c450 2023-08-25T02:28:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2664513239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 msgr2=0x559ea6466bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe694013710 tx=0x7fe694011000 comp rx=0 tx=0).stop 2023-08-25T02:28:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2664513239 shutdown_connections 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646ff20 0x559ea647a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646c450 0x559ea6466bd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/2664513239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ce10 0x559ea6476060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.389+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2664513239 >> 172.21.15.89:0/2664513239 conn(0x559ea63e1080 msgr2=0x559ea646bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2664513239 shutdown_connections 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 -- 172.21.15.89:0/2664513239 wait complete. 2023-08-25T02:28:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 -- start start 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 Processor -- start 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.390+0000 7fe6ac77db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea64fad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea647f180 con 0x559ea646ce10 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea64fb280 con 0x559ea646c450 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe6ac77db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea64fb3c0 con 0x559ea646ff20 2023-08-25T02:28:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe6ac77db80 10 client.0 _ll_register_callbacks cb 0x7fe69000b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48670/0 (socket says 172.21.15.89:48670) 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 -- 172.21.15.89:0/3848031445 learned_addr learned my addr 172.21.15.89:0/3848031445 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 -- 172.21.15.89:0/3848031445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 msgr2=0x559ea64fad40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea64fad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 -- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 msgr2=0x559ea647e330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T02:28:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 -- 172.21.15.89:0/3848031445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69401b040 con 0x559ea646ce10 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe69db74700 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe6880c8d40 tx=0x7fe6880982a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93bec6e1552632c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe681ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688098c70 con 0x559ea646ce10 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe6880174d0 con 0x559ea646ce10 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.391+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ea645f600 con 0x559ea646ce10 2023-08-25T02:28:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe68809c640 con 0x559ea646ce10 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 10 client.15453 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559ea61d3c00 con 0x559ea646ce10 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559ea645f600 con 0x559ea646ce10 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fe6880a0190 con 0x559ea646ce10 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 10 client.15453 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 20 client.15453 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:05.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64fcbc0 0x559ea64fefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 4 client.15453 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:28:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.392+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559ea63b7910 con 0x559ea64fcbc0 2023-08-25T02:28:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.393+0000 7fe69e375700 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64fcbc0 0x559ea64fefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.397+0000 7fe69e375700 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64fcbc0 0x559ea64fefe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e396d47df859a202 server_cookie=78287d00f3ad68c9 in_seq=0 out_seq=0 2023-08-25T02:28:05.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.397+0000 7fe681ffb700 10 client.15453 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:05.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe681ffb700 1 -- 172.21.15.89:0/3848031445 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559ea63b7910 con 0x559ea64fcbc0 2023-08-25T02:28:05.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe681ffb700 10 client.15453 handle_command_reply: tid=0 2023-08-25T02:28:05.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 client.15453 shutdown 2023-08-25T02:28:05.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 msgr2=0x559ea647e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe6880c8d40 tx=0x7fe6880982a0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 shutdown_connections 2023-08-25T02:28:05.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559ea64fcbc0 0x559ea64fefe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ea646c450 0x559ea647e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559ea646ff20 0x559ea64fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 --2- 172.21.15.89:0/3848031445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ea646ce10 0x559ea647e870 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe681ffb700 0 client.15453 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 >> 172.21.15.89:0/3848031445 conn(0x559ea63e1080 msgr2=0x559ea646b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 shutdown_connections 2023-08-25T02:28:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.686+0000 7fe6ac77db80 1 -- 172.21.15.89:0/3848031445 wait complete. 2023-08-25T02:28:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6ac77db80 20 client.15453 trim_cache size 0 max 16384 2023-08-25T02:28:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:28:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:28:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:28:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:28:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 msgr2=0x7fe6840a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 0x7fe6840a2600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe694009730 tx=0x7fe694006cb0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 msgr2=0x7fe69819cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe68800b530 tx=0x7fe68800b8f0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 shutdown_connections 2023-08-25T02:28:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe6840a0140 0x7fe6840a2600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6980ff9b0 0x7fe69819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe698105800 0x7fe698196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 --2- 172.21.15.89:0/241277827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe698100370 0x7fe69819cff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 >> 172.21.15.89:0/241277827 conn(0x7fe698077750 msgr2=0x7fe698103500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 shutdown_connections 2023-08-25T02:28:05.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:05.687+0000 7fe6a05d9700 1 -- 172.21.15.89:0/241277827 wait complete. 2023-08-25T02:28:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:05 smithi089 ceph-mon[98732]: pgmap v50: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 83 op/s 2023-08-25T02:28:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:05 smithi121 ceph-mon[100099]: pgmap v50: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 83 op/s 2023-08-25T02:28:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:05 smithi122 ceph-mon[100457]: pgmap v50: 97 pgs: 97 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.8 MiB/s wr, 83 op/s 2023-08-25T02:28:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:07 smithi089 ceph-mon[98732]: pgmap v51: 97 pgs: 97 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 101 op/s 2023-08-25T02:28:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:07 smithi121 ceph-mon[100099]: pgmap v51: 97 pgs: 97 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 101 op/s 2023-08-25T02:28:07.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:07 smithi122 ceph-mon[100457]: pgmap v51: 97 pgs: 97 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 101 op/s 2023-08-25T02:28:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:09 smithi089 ceph-mon[98732]: pgmap v52: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 MiB/s wr, 50 op/s 2023-08-25T02:28:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:09 smithi121 ceph-mon[100099]: pgmap v52: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 MiB/s wr, 50 op/s 2023-08-25T02:28:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:09 smithi122 ceph-mon[100457]: pgmap v52: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 MiB/s wr, 50 op/s 2023-08-25T02:28:10.201 INFO:teuthology.orchestra.run.smithi089.stderr:41000000 of 50000000 tuples (82%) done (elapsed 89.50 s, remaining 19.65 s) 2023-08-25T02:28:11.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:11 smithi121 ceph-mon[100099]: pgmap v53: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 MiB/s wr, 38 op/s 2023-08-25T02:28:11.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:11 smithi122 ceph-mon[100457]: pgmap v53: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 MiB/s wr, 38 op/s 2023-08-25T02:28:11.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:11 smithi089 ceph-mon[98732]: pgmap v53: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 33 MiB/s wr, 38 op/s 2023-08-25T02:28:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:41100000 of 50000000 tuples (82%) done (elapsed 92.12 s, remaining 19.95 s) 2023-08-25T02:28:12.925 INFO:teuthology.orchestra.run.smithi089.stderr:41200000 of 50000000 tuples (82%) done (elapsed 92.22 s, remaining 19.70 s) 2023-08-25T02:28:13.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:13 smithi121 ceph-mon[100099]: pgmap v54: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 350 op/s 2023-08-25T02:28:13.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:13 smithi122 ceph-mon[100457]: pgmap v54: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 350 op/s 2023-08-25T02:28:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:13 smithi089 ceph-mon[98732]: pgmap v54: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 350 op/s 2023-08-25T02:28:15.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:15 smithi121 ceph-mon[100099]: pgmap v55: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 331 op/s 2023-08-25T02:28:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:15 smithi122 ceph-mon[100457]: pgmap v55: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 331 op/s 2023-08-25T02:28:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:15 smithi089 ceph-mon[98732]: pgmap v55: 97 pgs: 97 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 331 op/s 2023-08-25T02:28:17.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:17 smithi121 ceph-mon[100099]: pgmap v56: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 372 op/s 2023-08-25T02:28:17.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:17 smithi122 ceph-mon[100457]: pgmap v56: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 372 op/s 2023-08-25T02:28:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:17 smithi089 ceph-mon[98732]: pgmap v56: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 36 MiB/s wr, 372 op/s 2023-08-25T02:28:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 -- 172.21.15.89:0/83267715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c100190 msgr2=0x7fc13c10c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 --2- 172.21.15.89:0/83267715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c100190 0x7fc13c10c6a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc13400b0a0 tx=0x7fc13400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 -- 172.21.15.89:0/83267715 shutdown_connections 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 --2- 172.21.15.89:0/83267715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c100190 0x7fc13c10c6a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 --2- 172.21.15.89:0/83267715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c1057e0 0x7fc13c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 --2- 172.21.15.89:0/83267715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c0ff7d0 0x7fc13c0ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 -- 172.21.15.89:0/83267715 >> 172.21.15.89:0/83267715 conn(0x7fc13c0fb260 msgr2=0x7fc13c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 -- 172.21.15.89:0/83267715 shutdown_connections 2023-08-25T02:28:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 -- 172.21.15.89:0/83267715 wait complete. 2023-08-25T02:28:18.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.804+0000 7fc1410b0700 1 Processor -- start 2023-08-25T02:28:18.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 -- start start 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c0ff7d0 0x7fc13c19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c100190 0x7fc13c19bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c195c60 con 0x7fc13c1057e0 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c195dd0 con 0x7fc13c100190 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc1410b0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c195f90 con 0x7fc13c0ff7d0 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48684/0 (socket says 172.21.15.89:48684) 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 -- 172.21.15.89:0/3909647100 learned_addr learned my addr 172.21.15.89:0/3909647100 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c100190 0x7fc13c19bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13ad9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c0ff7d0 0x7fc13c19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 -- 172.21.15.89:0/3909647100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c0ff7d0 msgr2=0x7fc13c19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c0ff7d0 0x7fc13c19b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.810 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 -- 172.21.15.89:0/3909647100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c100190 msgr2=0x7fc13c19bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c100190 0x7fc13c19bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.805+0000 7fc13b59e700 1 -- 172.21.15.89:0/3909647100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc134009d00 con 0x7fc13c1057e0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc13400b780 tx=0x7fc134007640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc123fff700 1 -- 172.21.15.89:0/3909647100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13400e040 con 0x7fc13c1057e0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc123fff700 1 -- 172.21.15.89:0/3909647100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc13401e070 con 0x7fc13c1057e0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc123fff700 1 -- 172.21.15.89:0/3909647100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1340052a0 con 0x7fc13c1057e0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c196210 con 0x7fc13c1057e0 2023-08-25T02:28:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.806+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c068a10 con 0x7fc13c1057e0 2023-08-25T02:28:18.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.808+0000 7fc123fff700 1 -- 172.21.15.89:0/3909647100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fc13401a4c0 con 0x7fc13c1057e0 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.808+0000 7fc123fff700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 0x7fc1240a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.808+0000 7fc123fff700 1 -- 172.21.15.89:0/3909647100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fc134020070 con 0x7fc13c1057e0 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.809+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 0x7fc1240a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.809+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 0x7fc1240a2580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009be0 tx=0x7fc12c009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.809+0000 7fc14d254b80 1 Processor -- start 2023-08-25T02:28:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 -- start start 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae3f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae39dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ada1750 con 0x56300ae3f860 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ad92720 con 0x56300ae39990 2023-08-25T02:28:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc14d254b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ad90040 con 0x56300ae3ef40 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48690/0 (socket says 172.21.15.89:48690) 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 -- 172.21.15.89:0/3470048844 learned_addr learned my addr 172.21.15.89:0/3470048844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae3f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae39dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 -- 172.21.15.89:0/3470048844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 msgr2=0x56300ae3f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae3f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 -- 172.21.15.89:0/3470048844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 msgr2=0x56300ae39dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae39dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 -- 172.21.15.89:0/3470048844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56300ac0b5d0 con 0x56300ae3f860 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae39dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.810+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc130007c90 tx=0x7fc1300103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c248e65ef6f88c3d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc1217fa700 1 -- 172.21.15.89:0/3470048844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130010ca0 con 0x56300ae3f860 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc1217fa700 1 -- 172.21.15.89:0/3470048844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc130004510 con 0x56300ae3f860 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc1217fa700 1 -- 172.21.15.89:0/3470048844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13001a470 con 0x56300ae3f860 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 -- 172.21.15.89:0/3470048844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 msgr2=0x56300ae39450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc130007c90 tx=0x7fc1300103d0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 -- 172.21.15.89:0/3470048844 shutdown_connections 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae39dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae3f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3470048844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3f860 0x56300ae39450 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 -- 172.21.15.89:0/3470048844 >> 172.21.15.89:0/3470048844 conn(0x56300adae080 msgr2=0x56300ae33d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 -- 172.21.15.89:0/3470048844 shutdown_connections 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.811+0000 7fc14d254b80 1 -- 172.21.15.89:0/3470048844 wait complete. 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 -- start start 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 Processor -- start 2023-08-25T02:28:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 0x56300ae42ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ad8b930 con 0x56300ae39990 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ad29b50 con 0x56300ae3f860 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ad29850 con 0x56300ae3ef40 2023-08-25T02:28:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc14d254b80 10 client.0 _ll_register_callbacks cb 0x56300ac73a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48694/0 (socket says 172.21.15.89:48694) 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13ad9d700 1 -- 172.21.15.89:0/3098242834 learned_addr learned my addr 172.21.15.89:0/3098242834 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 0x56300ae42ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.812+0000 7fc13ad9d700 1 -- 172.21.15.89:0/3098242834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 msgr2=0x56300ae428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13ad9d700 1 -- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 msgr2=0x56300ae42ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 0x56300ae42ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13ad9d700 1 -- 172.21.15.89:0/3098242834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1300078c0 con 0x56300ae39990 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13b59e700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 0x56300ae42ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc12c01fbf0 tx=0x7fc12c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd63db0377f37cae server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c011e30 con 0x56300ae39990 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56300ae09160 con 0x56300ae39990 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fc12c004020 con 0x56300ae39990 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc12c004220 con 0x56300ae39990 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 10 client.15465 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56300abb7d80 con 0x56300ae39990 2023-08-25T02:28:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56300ae09160 con 0x56300ae39990 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc12c0038d0 con 0x56300ae39990 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 10 client.15465 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 10 client.15465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 20 client.15465 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300ae55640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 4 client.15465 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:28:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.813+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56300ad31b40 con 0x56300ae53220 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.814+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300ae55640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.814+0000 7fc11eff5700 10 client.15465 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.814+0000 7fc13a59c700 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300ae55640 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e07800204ad6da0 server_cookie=569b9bdca9be8452 in_seq=0 out_seq=0 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc11eff5700 1 -- 172.21.15.89:0/3098242834 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56300ad31b40 con 0x56300ae53220 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc11eff5700 10 client.15465 handle_command_reply: tid=0 2023-08-25T02:28:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 client.15465 shutdown 2023-08-25T02:28:18.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 msgr2=0x56300ae45640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc12c01fbf0 tx=0x7fc12c0047c0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 shutdown_connections 2023-08-25T02:28:18.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300ae55640 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3f860 0x56300ae42ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc11eff5700 0 client.15465 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae45640 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 --2- 172.21.15.89:0/3098242834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3ef40 0x56300ae428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 >> 172.21.15.89:0/3098242834 conn(0x56300adae080 msgr2=0x56300ae2e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 shutdown_connections 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 1 -- 172.21.15.89:0/3098242834 wait complete. 2023-08-25T02:28:18.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.815+0000 7fc14d254b80 20 client.15465 trim_cache size 0 max 16384 2023-08-25T02:28:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.829+0000 7fc14d254b80 1 Processor -- start 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.829+0000 7fc14d254b80 1 -- start start 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 0x56300ae4a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae4f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ac0d8c0 con 0x56300ae39990 2023-08-25T02:28:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ae43e70 con 0x56300ae3ef40 2023-08-25T02:28:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc14d254b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ae2c390 con 0x56300ae3f860 2023-08-25T02:28:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48702/0 (socket says 172.21.15.89:48702) 2023-08-25T02:28:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 -- 172.21.15.89:0/1659702783 learned_addr learned my addr 172.21.15.89:0/1659702783 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13b59e700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae4f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13a59c700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 0x56300ae4a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 -- 172.21.15.89:0/1659702783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 msgr2=0x56300ae4f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae4f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 -- 172.21.15.89:0/1659702783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 msgr2=0x56300ae4a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 0x56300ae4a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 -- 172.21.15.89:0/1659702783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56300aba0c00 con 0x56300ae39990 2023-08-25T02:28:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13a59c700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 0x56300ae4a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc12c016040 tx=0x7fc12c011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b610804912909b9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc121ffb700 1 -- 172.21.15.89:0/1659702783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c003740 con 0x56300ae39990 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc121ffb700 1 -- 172.21.15.89:0/1659702783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc12c0038a0 con 0x56300ae39990 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.830+0000 7fc121ffb700 1 -- 172.21.15.89:0/1659702783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c004180 con 0x56300ae39990 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- 172.21.15.89:0/1659702783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 msgr2=0x56300ae39d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc12c016040 tx=0x7fc12c011b10 comp rx=0 tx=0).stop 2023-08-25T02:28:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- 172.21.15.89:0/1659702783 shutdown_connections 2023-08-25T02:28:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae4f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae39990 0x56300ae39d70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1659702783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae3ef40 0x56300ae4a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- 172.21.15.89:0/1659702783 >> 172.21.15.89:0/1659702783 conn(0x56300adae080 msgr2=0x56300ae33c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- 172.21.15.89:0/1659702783 shutdown_connections 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- 172.21.15.89:0/1659702783 wait complete. 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 -- start start 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 Processor -- start 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae3a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.831+0000 7fc14d254b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae3b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ae3baf0 con 0x56300ae3ef40 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ae3bc60 con 0x56300ae39990 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300ae3be20 con 0x56300ae3f860 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 10 client.0 _ll_register_callbacks cb 0x7fc12c01f7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48710/0 (socket says 172.21.15.89:48710) 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 -- 172.21.15.89:0/1161047984 learned_addr learned my addr 172.21.15.89:0/1161047984 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13b59e700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae3b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae3a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 -- 172.21.15.89:0/1161047984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 msgr2=0x56300ae3b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae3b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 -- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 msgr2=0x56300ae3a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae3a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 -- 172.21.15.89:0/1161047984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56300abb7d80 con 0x56300ae3ef40 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae3a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc13a59c700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fa0 tx=0x7fc130003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8e182dc174f474 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc11eff5700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13001f040 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fc130010aa0 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56300ae2c600 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc130010c60 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 10 client.15471 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.832+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56300abde1f0 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56300ae2c600 con 0x56300ae3ef40 2023-08-25T02:28:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc130005290 con 0x56300ae3ef40 2023-08-25T02:28:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 20 client.15471 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300aea9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 4 client.15471 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:28:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56300ad84910 con 0x56300ae53220 2023-08-25T02:28:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300aea9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc11eff5700 10 client.15471 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:18.833+0000 7fc13ad9d700 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300aea9e40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c4eb57375b3079 server_cookie=447af05934885191 in_seq=0 out_seq=0 2023-08-25T02:28:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc11eff5700 1 -- 172.21.15.89:0/1161047984 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56300ad84910 con 0x56300ae53220 2023-08-25T02:28:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc11eff5700 10 client.15471 handle_command_reply: tid=0 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 client.15471 shutdown 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 msgr2=0x56300ae3adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fa0 tx=0x7fc130003680 comp rx=0 tx=0).stop 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 shutdown_connections 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x56300ae53220 0x56300aea9e40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56300ae39990 0x56300ae3a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56300ae3f860 0x56300ae3b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 --2- 172.21.15.89:0/1161047984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56300ae3ef40 0x56300ae3adc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc11eff5700 0 client.15471 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 >> 172.21.15.89:0/1161047984 conn(0x56300adae080 msgr2=0x56300ae4f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 shutdown_connections 2023-08-25T02:28:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 1 -- 172.21.15.89:0/1161047984 wait complete. 2023-08-25T02:28:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.187+0000 7fc14d254b80 20 client.15471 trim_cache size 0 max 16384 2023-08-25T02:28:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:28:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:28:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:28:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:28:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 msgr2=0x7fc1240a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 0x7fc1240a2580 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009be0 tx=0x7fc12c009260 comp rx=0 tx=0).stop 2023-08-25T02:28:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 msgr2=0x7fc13c1955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc13400b780 tx=0x7fc134007640 comp rx=0 tx=0).stop 2023-08-25T02:28:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 shutdown_connections 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fc1240a00c0 0x7fc1240a2580 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc13c1057e0 0x7fc13c1955f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc13c0ff7d0 0x7fc13c19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 --2- 172.21.15.89:0/3909647100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c100190 0x7fc13c19bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 >> 172.21.15.89:0/3909647100 conn(0x7fc13c0fb260 msgr2=0x7fc13c0fb790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 shutdown_connections 2023-08-25T02:28:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:19.188+0000 7fc1410b0700 1 -- 172.21.15.89:0/3909647100 wait complete. 2023-08-25T02:28:19.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:19 smithi121 ceph-mon[100099]: pgmap v57: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.3 MiB/s wr, 354 op/s 2023-08-25T02:28:19.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:19 smithi122 ceph-mon[100457]: pgmap v57: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.3 MiB/s wr, 354 op/s 2023-08-25T02:28:19.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:19 smithi089 ceph-mon[98732]: pgmap v57: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.3 MiB/s wr, 354 op/s 2023-08-25T02:28:21.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:21 smithi121 ceph-mon[100099]: pgmap v58: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 353 op/s 2023-08-25T02:28:21.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:21 smithi122 ceph-mon[100457]: pgmap v58: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 353 op/s 2023-08-25T02:28:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:21 smithi089 ceph-mon[98732]: pgmap v58: 97 pgs: 97 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 353 op/s 2023-08-25T02:28:23.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:23 smithi121 ceph-mon[100099]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 MiB/s wr, 364 op/s 2023-08-25T02:28:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:23 smithi122 ceph-mon[100457]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 MiB/s wr, 364 op/s 2023-08-25T02:28:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:23 smithi089 ceph-mon[98732]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 MiB/s wr, 364 op/s 2023-08-25T02:28:25.461 INFO:teuthology.orchestra.run.smithi089.stderr:41300000 of 50000000 tuples (82%) done (elapsed 104.76 s, remaining 22.07 s) 2023-08-25T02:28:25.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:25 smithi121 ceph-mon[100099]: pgmap v60: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 52 op/s 2023-08-25T02:28:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:25 smithi122 ceph-mon[100457]: pgmap v60: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 52 op/s 2023-08-25T02:28:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:25 smithi089 ceph-mon[98732]: pgmap v60: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 52 op/s 2023-08-25T02:28:27.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:27 smithi121 ceph-mon[100099]: pgmap v61: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 90 op/s 2023-08-25T02:28:27.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:27 smithi122 ceph-mon[100457]: pgmap v61: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 90 op/s 2023-08-25T02:28:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:27 smithi089 ceph-mon[98732]: pgmap v61: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 MiB/s wr, 90 op/s 2023-08-25T02:28:29.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:29 smithi122 ceph-mon[100457]: pgmap v62: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 49 op/s 2023-08-25T02:28:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:29 smithi089 ceph-mon[98732]: pgmap v62: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 49 op/s 2023-08-25T02:28:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:29 smithi121 ceph-mon[100099]: pgmap v62: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 49 op/s 2023-08-25T02:28:31.003 INFO:teuthology.orchestra.run.smithi089.stderr:41400000 of 50000000 tuples (82%) done (elapsed 110.30 s, remaining 22.91 s) 2023-08-25T02:28:31.077 INFO:teuthology.orchestra.run.smithi089.stderr:41500000 of 50000000 tuples (83%) done (elapsed 110.37 s, remaining 22.61 s) 2023-08-25T02:28:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:31 smithi121 ceph-mon[100099]: pgmap v63: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 MiB/s wr, 49 op/s 2023-08-25T02:28:31.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:31 smithi122 ceph-mon[100457]: pgmap v63: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 MiB/s wr, 49 op/s 2023-08-25T02:28:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:31 smithi089 ceph-mon[98732]: pgmap v63: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 MiB/s wr, 49 op/s 2023-08-25T02:28:32.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 -- 172.21.15.89:0/2277705286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 msgr2=0x7fad8410c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/2277705286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad8410c700 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00b0a0 tx=0x7fad7c00b3b0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 -- 172.21.15.89:0/2277705286 shutdown_connections 2023-08-25T02:28:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/2277705286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad8410c700 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/2277705286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 0x7fad84105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/2277705286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad84100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 -- 172.21.15.89:0/2277705286 >> 172.21.15.89:0/2277705286 conn(0x7fad84075ce0 msgr2=0x7fad840fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 -- 172.21.15.89:0/2277705286 shutdown_connections 2023-08-25T02:28:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.310+0000 7fad8c2b9700 1 -- 172.21.15.89:0/2277705286 wait complete. 2023-08-25T02:28:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 Processor -- start 2023-08-25T02:28:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 -- start start 2023-08-25T02:28:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad840fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 0x7fad840fbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad841a93b0 con 0x7fad84100850 2023-08-25T02:28:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad841a9520 con 0x7fad840ffe90 2023-08-25T02:28:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8c2b9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad841a96e0 con 0x7fad84105840 2023-08-25T02:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48724/0 (socket says 172.21.15.89:48724) 2023-08-25T02:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad89854700 1 -- 172.21.15.89:0/4256633984 learned_addr learned my addr 172.21.15.89:0/4256633984 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8a856700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 0x7fad840fbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.311+0000 7fad8a055700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad840fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 -- 172.21.15.89:0/4256633984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 msgr2=0x7fad840fbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 0x7fad840fbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 -- 172.21.15.89:0/4256633984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 msgr2=0x7fad840fb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad840fb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 -- 172.21.15.89:0/4256633984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c009d00 con 0x7fad84100850 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad8a055700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad840fb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad89854700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fad8000d950 tx=0x7fad8000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad7b7fe700 1 -- 172.21.15.89:0/4256633984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80009940 con 0x7fad84100850 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad7b7fe700 1 -- 172.21.15.89:0/4256633984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad80010460 con 0x7fad84100850 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad841a99c0 con 0x7fad84100850 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad7b7fe700 1 -- 172.21.15.89:0/4256633984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8000f620 con 0x7fad84100850 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.312+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad841a9ed0 con 0x7fad84100850 2023-08-25T02:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.314+0000 7fad7b7fe700 1 -- 172.21.15.89:0/4256633984 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fad800108d0 con 0x7fad84100850 2023-08-25T02:28:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.314+0000 7fad7b7fe700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 0x7fad700a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.315+0000 7fad8a055700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 0x7fad700a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.315+0000 7fad7b7fe700 1 -- 172.21.15.89:0/4256633984 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fad800cb4b0 con 0x7fad84100850 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.315+0000 7fad8a055700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 0x7fad700a2580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad74009c90 tx=0x7fad74009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.315+0000 7fad9845db80 1 Processor -- start 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 -- start start 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556becd7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bec39750 con 0x5556becd7860 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bec2a720 con 0x5556becd6f40 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad9845db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bec28040 con 0x5556becd1990 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48732/0 (socket says 172.21.15.89:48732) 2023-08-25T02:28:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 -- 172.21.15.89:0/3252721172 learned_addr learned my addr 172.21.15.89:0/3252721172 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad8a856700 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad8a055700 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556becd7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 -- 172.21.15.89:0/3252721172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 msgr2=0x5556becd1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 -- 172.21.15.89:0/3252721172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 msgr2=0x5556becd7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556becd7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 -- 172.21.15.89:0/3252721172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556beaa35d0 con 0x5556becd7860 2023-08-25T02:28:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad89854700 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fad800a04a0 tx=0x7fad800992c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f6c89afffeafa8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.316+0000 7fad78da2700 1 -- 172.21.15.89:0/3252721172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8009c9f0 con 0x5556becd7860 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad78da2700 1 -- 172.21.15.89:0/3252721172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad800a09d0 con 0x5556becd7860 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad78da2700 1 -- 172.21.15.89:0/3252721172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80098310 con 0x5556becd7860 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 -- 172.21.15.89:0/3252721172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 msgr2=0x5556becd1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fad800a04a0 tx=0x7fad800992c0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 -- 172.21.15.89:0/3252721172 shutdown_connections 2023-08-25T02:28:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556becd7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 --2- 172.21.15.89:0/3252721172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd7860 0x5556becd1450 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 -- 172.21.15.89:0/3252721172 >> 172.21.15.89:0/3252721172 conn(0x5556bec46080 msgr2=0x5556beccbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 -- 172.21.15.89:0/3252721172 shutdown_connections 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.317+0000 7fad9845db80 1 -- 172.21.15.89:0/3252721172 wait complete. 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 -- start start 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 Processor -- start 2023-08-25T02:28:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 0x5556becdd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bec23930 con 0x5556becd6f40 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bebc1b50 con 0x5556becd1990 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556bebc1850 con 0x5556becd7860 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad9845db80 10 client.0 _ll_register_callbacks cb 0x5556beb0ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48738/0 (socket says 172.21.15.89:48738) 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad89854700 1 -- 172.21.15.89:0/2388865601 learned_addr learned my addr 172.21.15.89:0/2388865601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.318+0000 7fad8a856700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad8a055700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 0x5556becdd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 -- 172.21.15.89:0/2388865601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 msgr2=0x5556becdb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 -- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 msgr2=0x5556becdd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 0x5556becdd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 -- 172.21.15.89:0/2388865601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad841a9ed0 con 0x5556becd6f40 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad8a055700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 0x5556becdd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad89854700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad800c8cb0 tx=0x7fad80010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3537009207319a16 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8009c9f0 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fad800982c0 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556beca1160 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad80098e20 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 10 client.15483 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5556bea4fd80 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad80062080 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fad800a05b0 con 0x5556becd6f40 2023-08-25T02:28:32.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 10 client.15483 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 10 client.15483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 20 client.15483 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.319+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556beceace0 0x5556beced0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.320+0000 7fad9845db80 4 client.15483 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.320+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556bebc9b40 con 0x5556beceace0 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.320+0000 7fad8a055700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556beceace0 0x5556beced0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.320+0000 7fad8a055700 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556beceace0 0x5556beced0d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a693b196ba94110f server_cookie=fa26041c431b9877 in_seq=0 out_seq=0 2023-08-25T02:28:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.320+0000 7fad6dffb700 10 client.15483 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad6dffb700 1 -- 172.21.15.89:0/2388865601 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5556bebc9b40 con 0x5556beceace0 2023-08-25T02:28:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad6dffb700 10 client.15483 handle_command_reply: tid=0 2023-08-25T02:28:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 client.15483 shutdown 2023-08-25T02:28:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 msgr2=0x5556becda950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad800c8cb0 tx=0x7fad80010cd0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 shutdown_connections 2023-08-25T02:28:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556beceace0 0x5556beced0d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd1990 0x5556becdd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 --2- 172.21.15.89:0/2388865601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becda950 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 >> 172.21.15.89:0/2388865601 conn(0x5556bec46080 msgr2=0x5556becc6820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad6dffb700 0 client.15483 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 shutdown_connections 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 1 -- 172.21.15.89:0/2388865601 wait complete. 2023-08-25T02:28:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.321+0000 7fad9845db80 20 client.15483 trim_cache size 0 max 16384 2023-08-25T02:28:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 Processor -- start 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 -- start start 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556bece2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556beaa58c0 con 0x5556becd1990 2023-08-25T02:28:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556becde060 con 0x5556becd6f40 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad9845db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556beccc8f0 con 0x5556becd7860 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48744/0 (socket says 172.21.15.89:48744) 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 -- 172.21.15.89:0/3319499012 learned_addr learned my addr 172.21.15.89:0/3319499012 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a856700 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad89854700 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556bece2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 -- 172.21.15.89:0/3319499012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 msgr2=0x5556becdd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 -- 172.21.15.89:0/3319499012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 msgr2=0x5556bece2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556bece2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 -- 172.21.15.89:0/3319499012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556becd2040 con 0x5556becd1990 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad8a055700 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fad74010a40 tx=0x7fad74013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1272590bda52e7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad795a3700 1 -- 172.21.15.89:0/3319499012 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74025040 con 0x5556becd1990 2023-08-25T02:28:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad795a3700 1 -- 172.21.15.89:0/3319499012 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad74015070 con 0x5556becd1990 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.336+0000 7fad795a3700 1 -- 172.21.15.89:0/3319499012 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad740116a0 con 0x5556becd1990 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- 172.21.15.89:0/3319499012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 msgr2=0x5556becd1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fad74010a40 tx=0x7fad74013000 comp rx=0 tx=0).stop 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- 172.21.15.89:0/3319499012 shutdown_connections 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd7860 0x5556becdd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd1990 0x5556becd1450 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 --2- 172.21.15.89:0/3319499012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd6f40 0x5556bece2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- 172.21.15.89:0/3319499012 >> 172.21.15.89:0/3319499012 conn(0x5556bec46080 msgr2=0x5556bece69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- 172.21.15.89:0/3319499012 shutdown_connections 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- 172.21.15.89:0/3319499012 wait complete. 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 -- start start 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.337+0000 7fad9845db80 1 Processor -- start 2023-08-25T02:28:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd7860 0x5556becd34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556becd3c00 con 0x5556becd6f40 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556becd3d70 con 0x5556becd7860 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556becd3f30 con 0x5556becd1990 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 10 client.0 _ll_register_callbacks cb 0x7fad74005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48752/0 (socket says 172.21.15.89:48752) 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad8a055700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 -- 172.21.15.89:0/379451616 learned_addr learned my addr 172.21.15.89:0/379451616 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad8a856700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd7860 0x5556becd34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 -- 172.21.15.89:0/379451616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 msgr2=0x5556becd2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 -- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd7860 msgr2=0x5556becd34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd7860 0x5556becd34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 -- 172.21.15.89:0/379451616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad74010c50 con 0x5556becd6f40 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad8a055700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad89854700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fad800950c0 tx=0x7fad80098700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8e827c8df9c935 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad6dffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad800cb310 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fad8009c9f0 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.338+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556becc4600 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad800992c0 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 10 client.15489 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5556bea38c00 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5556becc4600 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fad80098e20 con 0x5556becd6f40 2023-08-25T02:28:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 20 client.15489 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556bed3f500 0x5556bed41920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 4 client.15489 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5556bec1c910 con 0x5556bed3f500 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.339+0000 7fad8a055700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556bed3f500 0x5556bed41920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.340+0000 7fad6dffb700 10 client.15489 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:32.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.340+0000 7fad8a055700 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556bed3f500 0x5556bed41920 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4423e3620fbe3a1 server_cookie=4651d00fb11f61b5 in_seq=0 out_seq=0 2023-08-25T02:28:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad6dffb700 1 -- 172.21.15.89:0/379451616 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5556bec1c910 con 0x5556bed3f500 2023-08-25T02:28:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad6dffb700 10 client.15489 handle_command_reply: tid=0 2023-08-25T02:28:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 client.15489 shutdown 2023-08-25T02:28:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 msgr2=0x5556becd2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fad800950c0 tx=0x7fad80098700 comp rx=0 tx=0).stop 2023-08-25T02:28:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 shutdown_connections 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5556bed3f500 0x5556bed41920 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5556becd7860 0x5556becd34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad6dffb700 0 client.15489 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556becd1990 0x5556becd2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 --2- 172.21.15.89:0/379451616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556becd6f40 0x5556becd2fb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 >> 172.21.15.89:0/379451616 conn(0x5556bec46080 msgr2=0x5556becd4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 shutdown_connections 2023-08-25T02:28:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 1 -- 172.21.15.89:0/379451616 wait complete. 2023-08-25T02:28:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.604+0000 7fad9845db80 20 client.15489 trim_cache size 0 max 16384 2023-08-25T02:28:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:28:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:28:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:28:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:28:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 msgr2=0x7fad700a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 0x7fad700a2580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad74009c90 tx=0x7fad74009260 comp rx=0 tx=0).stop 2023-08-25T02:28:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 msgr2=0x7fad840fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fad8000d950 tx=0x7fad8000dc60 comp rx=0 tx=0).stop 2023-08-25T02:28:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 shutdown_connections 2023-08-25T02:28:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fad700a00c0 0x7fad700a2580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad840ffe90 0x7fad840fb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fad84105840 0x7fad840fbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 --2- 172.21.15.89:0/4256633984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad84100850 0x7fad840fb630 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 >> 172.21.15.89:0/4256633984 conn(0x7fad84075ce0 msgr2=0x7fad840fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 shutdown_connections 2023-08-25T02:28:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:32.605+0000 7fad8c2b9700 1 -- 172.21.15.89:0/4256633984 wait complete. 2023-08-25T02:28:33.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:33 smithi121 ceph-mon[100099]: pgmap v64: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 55 op/s 2023-08-25T02:28:33.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:33 smithi122 ceph-mon[100457]: pgmap v64: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 55 op/s 2023-08-25T02:28:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:33 smithi089 ceph-mon[98732]: pgmap v64: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 MiB/s wr, 55 op/s 2023-08-25T02:28:35.674 INFO:teuthology.orchestra.run.smithi089.stderr:41600000 of 50000000 tuples (83%) done (elapsed 114.97 s, remaining 23.22 s) 2023-08-25T02:28:35.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:35 smithi121 ceph-mon[100099]: pgmap v65: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:28:35.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:35 smithi122 ceph-mon[100457]: pgmap v65: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:28:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:35 smithi089 ceph-mon[98732]: pgmap v65: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:28:37.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:37 smithi121 ceph-mon[100099]: pgmap v66: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 51 op/s 2023-08-25T02:28:37.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:37 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:28:37.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:37 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:28:37.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:37 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:28:37.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:37 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:28:37.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:37 smithi122 ceph-mon[100457]: pgmap v66: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 51 op/s 2023-08-25T02:28:37.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:37 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:28:37.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:37 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:28:37.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:37 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:28:37.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:37 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:28:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:37 smithi089 ceph-mon[98732]: pgmap v66: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.9 MiB/s wr, 51 op/s 2023-08-25T02:28:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:37 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:28:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:37 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:28:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:37 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:28:37.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:37 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:28:39.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:39 smithi122 ceph-mon[100457]: pgmap v67: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 13 op/s 2023-08-25T02:28:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:39 smithi089 ceph-mon[98732]: pgmap v67: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 13 op/s 2023-08-25T02:28:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:39 smithi121 ceph-mon[100099]: pgmap v67: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.7 MiB/s wr, 13 op/s 2023-08-25T02:28:41.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:41 smithi121 ceph-mon[100099]: pgmap v68: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 13 op/s 2023-08-25T02:28:41.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:41 smithi122 ceph-mon[100457]: pgmap v68: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 13 op/s 2023-08-25T02:28:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:41 smithi089 ceph-mon[98732]: pgmap v68: 97 pgs: 97 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 13 op/s 2023-08-25T02:28:42.652 INFO:teuthology.orchestra.run.smithi089.stderr:41700000 of 50000000 tuples (83%) done (elapsed 121.95 s, remaining 24.27 s) 2023-08-25T02:28:42.719 INFO:teuthology.orchestra.run.smithi089.stderr:41800000 of 50000000 tuples (83%) done (elapsed 122.02 s, remaining 23.94 s) 2023-08-25T02:28:43.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:43 smithi121 ceph-mon[100099]: pgmap v69: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:43.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:43 smithi122 ceph-mon[100457]: pgmap v69: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:43 smithi089 ceph-mon[98732]: pgmap v69: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:45.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:45 smithi121 ceph-mon[100099]: pgmap v70: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 -- 172.21.15.89:0/3233650921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 msgr2=0x7f807010c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 --2- 172.21.15.89:0/3233650921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807010c700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f806800b0d0 tx=0x7f806800b3e0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 -- 172.21.15.89:0/3233650921 shutdown_connections 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 --2- 172.21.15.89:0/3233650921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 0x7f8070100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 --2- 172.21.15.89:0/3233650921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807010c700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 --2- 172.21.15.89:0/3233650921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8070100af0 0x7f80701043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 -- 172.21.15.89:0/3233650921 >> 172.21.15.89:0/3233650921 conn(0x7f8070075ce0 msgr2=0x7f80700fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 -- 172.21.15.89:0/3233650921 shutdown_connections 2023-08-25T02:28:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.723+0000 7f80779db700 1 -- 172.21.15.89:0/3233650921 wait complete. 2023-08-25T02:28:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 Processor -- start 2023-08-25T02:28:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 -- start start 2023-08-25T02:28:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 0x7f807019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80701957b0 0x7f8070195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8070196230 con 0x7f8070104af0 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8070196370 con 0x7f8070100140 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f80779db700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8070196530 con 0x7f80701957b0 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48766/0 (socket says 172.21.15.89:48766) 2023-08-25T02:28:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 -- 172.21.15.89:0/3736370793 learned_addr learned my addr 172.21.15.89:0/3736370793 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8075f78700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80701957b0 0x7f8070195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8075777700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 0x7f807019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 -- 172.21.15.89:0/3736370793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80701957b0 msgr2=0x7f8070195c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80701957b0 0x7f8070195c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 -- 172.21.15.89:0/3736370793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 msgr2=0x7f807019b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 0x7f807019b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.724+0000 7f8074f76700 1 -- 172.21.15.89:0/3736370793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8068009d00 con 0x7f8070104af0 2023-08-25T02:28:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f8074f76700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9f0 tx=0x7f806c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f80667fc700 1 -- 172.21.15.89:0/3736370793 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c009750 con 0x7f8070104af0 2023-08-25T02:28:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f80667fc700 1 -- 172.21.15.89:0/3736370793 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f806c00c7c0 con 0x7f8070104af0 2023-08-25T02:28:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80701a27c0 con 0x7f8070104af0 2023-08-25T02:28:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f80667fc700 1 -- 172.21.15.89:0/3736370793 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0054d0 con 0x7f8070104af0 2023-08-25T02:28:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.725+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80701a2c80 con 0x7f8070104af0 2023-08-25T02:28:45.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:45 smithi122 ceph-mon[100457]: pgmap v70: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:45.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.727+0000 7f80667fc700 1 -- 172.21.15.89:0/3736370793 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f806c00c930 con 0x7f8070104af0 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.727+0000 7f80667fc700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 0x7f805c0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.728+0000 7f8075777700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 0x7f805c0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.728+0000 7f80667fc700 1 -- 172.21.15.89:0/3736370793 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f806c014070 con 0x7f8070104af0 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.728+0000 7f8075777700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 0x7f805c0a2580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8060009c60 tx=0x7f8060009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 Processor -- start 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 -- start start 2023-08-25T02:28:45.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfa8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637df10750 con 0x55637dfadf40 2023-08-25T02:28:45.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637df01720 con 0x55637dfa8990 2023-08-25T02:28:45.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637deff040 con 0x55637dfae860 2023-08-25T02:28:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48772/0 (socket says 172.21.15.89:48772) 2023-08-25T02:28:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 -- 172.21.15.89:0/3307742307 learned_addr learned my addr 172.21.15.89:0/3307742307 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8074f76700 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfa8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075f78700 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 -- 172.21.15.89:0/3307742307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 msgr2=0x55637dfa8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfa8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 -- 172.21.15.89:0/3307742307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 msgr2=0x55637dfa8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 -- 172.21.15.89:0/3307742307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55637dd7a5d0 con 0x55637dfadf40 2023-08-25T02:28:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f8075777700 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f806000e4e0 tx=0x7f8060014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4412d797520600 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.729+0000 7f805b7fe700 1 -- 172.21.15.89:0/3307742307 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806000ee00 con 0x55637dfadf40 2023-08-25T02:28:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f805b7fe700 1 -- 172.21.15.89:0/3307742307 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8060021070 con 0x55637dfadf40 2023-08-25T02:28:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f805b7fe700 1 -- 172.21.15.89:0/3307742307 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806001d7f0 con 0x55637dfadf40 2023-08-25T02:28:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3307742307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 msgr2=0x55637dfae320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f806000e4e0 tx=0x7f8060014000 comp rx=0 tx=0).stop 2023-08-25T02:28:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3307742307 shutdown_connections 2023-08-25T02:28:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfae320 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3307742307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfa8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3307742307 >> 172.21.15.89:0/3307742307 conn(0x55637df1d080 msgr2=0x55637dfa2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3307742307 shutdown_connections 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.730+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3307742307 wait complete. 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 -- start start 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 Processor -- start 2023-08-25T02:28:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.747 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.747 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637defa930 con 0x55637dfae860 2023-08-25T02:28:45.747 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637de98b50 con 0x55637dfa8990 2023-08-25T02:28:45.747 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637de98850 con 0x55637dfadf40 2023-08-25T02:28:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8083b7fb80 10 client.0 _ll_register_callbacks cb 0x55637dde2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8075f78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8074f76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8074f76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40716/0 (socket says 172.21.15.89:40716) 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8075f78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48782/0 (socket says 172.21.15.89:48782) 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8074f76700 1 -- 172.21.15.89:0/2330547141 learned_addr learned my addr 172.21.15.89:0/2330547141 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.731+0000 7f8075777700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 -- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 msgr2=0x55637dfb46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 -- 172.21.15.89:0/2330547141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 msgr2=0x55637dfb20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 -- 172.21.15.89:0/2330547141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8060017070 con 0x55637dfadf40 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8075f78700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8074f76700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f806c098c90 tx=0x7f806c099820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7c26da91c54fe20 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c099440 con 0x55637dfadf40 2023-08-25T02:28:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f806c0a0df0 con 0x55637dfadf40 2023-08-25T02:28:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55637df78160 con 0x55637dfadf40 2023-08-25T02:28:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f806c0984d0 con 0x55637dfadf40 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8083b7fb80 10 client.25123 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55637dd26d80 con 0x55637dfadf40 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f806c063080 con 0x55637dfadf40 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.732+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f806c004a30 con 0x55637dfadf40 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 10 client.25123 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 10 client.25123 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 20 client.25123 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637dfc21d0 0x55637dfc45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 4 client.25123 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55637dea0b40 con 0x55637dfc21d0 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8075777700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637dfc21d0 0x55637dfc45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8058ff9700 10 client.25123 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.733+0000 7f8075777700 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637dfc21d0 0x55637dfc45f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c2274d023e2b7e server_cookie=14c9e4ced27267d0 in_seq=0 out_seq=0 2023-08-25T02:28:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8058ff9700 1 -- 172.21.15.89:0/2330547141 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55637dea0b40 con 0x55637dfc21d0 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8058ff9700 10 client.25123 handle_command_reply: tid=0 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 client.25123 shutdown 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 msgr2=0x55637dfb1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f806c098c90 tx=0x7f806c099820 comp rx=0 tx=0).stop 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 shutdown_connections 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637dfc21d0 0x55637dfc45f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfb20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8058ff9700 0 client.25123 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2330547141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 >> 172.21.15.89:0/2330547141 conn(0x55637df1d080 msgr2=0x55637df9d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 shutdown_connections 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2330547141 wait complete. 2023-08-25T02:28:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.734+0000 7f8083b7fb80 20 client.25123 trim_cache size 0 max 16384 2023-08-25T02:28:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 Processor -- start 2023-08-25T02:28:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 -- start start 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfb4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dd7c8c0 con 0x55637dfadf40 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dfbf840 con 0x55637dfa8990 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dfa32a0 con 0x55637dfae860 2023-08-25T02:28:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48788/0 (socket says 172.21.15.89:48788) 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 -- 172.21.15.89:0/3525581552 learned_addr learned my addr 172.21.15.89:0/3525581552 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8075f78700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfb4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8075777700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 -- 172.21.15.89:0/3525581552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 msgr2=0x55637dfb4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfb4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 -- 172.21.15.89:0/3525581552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 msgr2=0x55637dfa8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8074f76700 1 -- 172.21.15.89:0/3525581552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55637dfa9040 con 0x55637dfadf40 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.749+0000 7f8075777700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8074f76700 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f806c098f70 tx=0x7f806c09cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa315e2595b1354 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f805bfff700 1 -- 172.21.15.89:0/3525581552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0a0df0 con 0x55637dfadf40 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f805bfff700 1 -- 172.21.15.89:0/3525581552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55637df78160 con 0x55637dfadf40 2023-08-25T02:28:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f805bfff700 1 -- 172.21.15.89:0/3525581552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0986f0 con 0x55637dfadf40 2023-08-25T02:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3525581552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 msgr2=0x55637dfb9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f806c098f70 tx=0x7f806c09cf90 comp rx=0 tx=0).stop 2023-08-25T02:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3525581552 shutdown_connections 2023-08-25T02:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfa8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfae860 0x55637dfb4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/3525581552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfadf40 0x55637dfb9240 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3525581552 >> 172.21.15.89:0/3525581552 conn(0x55637df1d080 msgr2=0x55637dfabad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3525581552 shutdown_connections 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.750+0000 7f8083b7fb80 1 -- 172.21.15.89:0/3525581552 wait complete. 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 -- start start 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 Processor -- start 2023-08-25T02:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dfb0580 con 0x55637dfae860 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dfb06f0 con 0x55637dfa8990 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637dfb08b0 con 0x55637dfadf40 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8083b7fb80 10 client.0 _ll_register_callbacks cb 0x7f806c005670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8075f78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8074f76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8075f78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48796/0 (socket says 172.21.15.89:48796) 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.751+0000 7f8075f78700 1 -- 172.21.15.89:0/2651257626 learned_addr learned my addr 172.21.15.89:0/2651257626 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075777700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 -- 172.21.15.89:0/2651257626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 msgr2=0x55637dfb1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 -- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 msgr2=0x55637dfb11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 -- 172.21.15.89:0/2651257626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55637dd26d80 con 0x55637dfae860 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8074f76700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075777700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8075f78700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8068015000 tx=0x7f806800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f4053345332e6e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806800e040 con 0x55637dfae860 2023-08-25T02:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f8068003c00 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55637df9b600 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f806800bea0 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 10 client.15507 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55637dd0fc00 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55637df9b600 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f806800b100 con 0x55637dfae860 2023-08-25T02:28:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 10 client.15507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 20 client.15507 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637e0170a0 0x55637e0194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 4 client.15507 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.752+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55637def3910 con 0x55637e0170a0 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.753+0000 7f8075777700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637e0170a0 0x55637e0194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.753+0000 7f8058ff9700 10 client.15507 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.753+0000 7f8075777700 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637e0170a0 0x55637e0194c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf5ba94bd829dd7 server_cookie=74e81e33928ed58c in_seq=0 out_seq=0 2023-08-25T02:28:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8058ff9700 1 -- 172.21.15.89:0/2651257626 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55637def3910 con 0x55637e0170a0 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8058ff9700 10 client.15507 handle_command_reply: tid=0 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 client.15507 shutdown 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 msgr2=0x55637dfafec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8068015000 tx=0x7f806800ba40 comp rx=0 tx=0).stop 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 shutdown_connections 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55637e0170a0 0x55637e0194c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55637dfa8990 0x55637dfb11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8058ff9700 0 client.15507 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55637dfae860 0x55637dfafec0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 --2- 172.21.15.89:0/2651257626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55637dfadf40 0x55637dfb1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 >> 172.21.15.89:0/2651257626 conn(0x55637df1d080 msgr2=0x55637dfaec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 shutdown_connections 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 1 -- 172.21.15.89:0/2651257626 wait complete. 2023-08-25T02:28:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.762+0000 7f8083b7fb80 20 client.15507 trim_cache size 0 max 16384 2023-08-25T02:28:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:28:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:28:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:28:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:28:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 msgr2=0x7f805c0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 0x7f805c0a2580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8060009c60 tx=0x7f8060009260 comp rx=0 tx=0).stop 2023-08-25T02:28:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 msgr2=0x7f807019be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f806c00e9f0 tx=0x7f806c00ed00 comp rx=0 tx=0).stop 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 shutdown_connections 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f805c0a00c0 0x7f805c0a2580 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8070100140 0x7f807019b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80701957b0 0x7f8070195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 --2- 172.21.15.89:0/3736370793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8070104af0 0x7f807019be90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 >> 172.21.15.89:0/3736370793 conn(0x7f8070075ce0 msgr2=0x7f8070103360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 shutdown_connections 2023-08-25T02:28:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:45.763+0000 7f80779db700 1 -- 172.21.15.89:0/3736370793 wait complete. 2023-08-25T02:28:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:45 smithi089 ceph-mon[98732]: pgmap v70: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:47.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:47 smithi122 ceph-mon[100457]: pgmap v71: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.4 MiB/s wr, 19 op/s 2023-08-25T02:28:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:47 smithi089 ceph-mon[98732]: pgmap v71: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.4 MiB/s wr, 19 op/s 2023-08-25T02:28:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:47 smithi121 ceph-mon[100099]: pgmap v71: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.4 MiB/s wr, 19 op/s 2023-08-25T02:28:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:41900000 of 50000000 tuples (83%) done (elapsed 128.93 s, remaining 24.93 s) 2023-08-25T02:28:49.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:49 smithi122 ceph-mon[100457]: pgmap v72: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 MiB/s wr, 12 op/s 2023-08-25T02:28:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:49 smithi089 ceph-mon[98732]: pgmap v72: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 MiB/s wr, 12 op/s 2023-08-25T02:28:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:49 smithi121 ceph-mon[100099]: pgmap v72: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.7 MiB/s wr, 12 op/s 2023-08-25T02:28:51.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:51 smithi121 ceph-mon[100099]: pgmap v73: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:51 smithi122 ceph-mon[100457]: pgmap v73: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:51 smithi089 ceph-mon[98732]: pgmap v73: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 12 op/s 2023-08-25T02:28:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:53 smithi121 ceph-mon[100099]: pgmap v74: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:53.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:53 smithi122 ceph-mon[100457]: pgmap v74: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:53 smithi089 ceph-mon[98732]: pgmap v74: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:28:55.052 INFO:teuthology.orchestra.run.smithi089.stderr:42000000 of 50000000 tuples (84%) done (elapsed 134.35 s, remaining 25.59 s) 2023-08-25T02:28:55.119 INFO:teuthology.orchestra.run.smithi089.stderr:42100000 of 50000000 tuples (84%) done (elapsed 134.42 s, remaining 25.22 s) 2023-08-25T02:28:55.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:55 smithi122 ceph-mon[100457]: pgmap v75: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 14 op/s 2023-08-25T02:28:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:55 smithi089 ceph-mon[98732]: pgmap v75: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 14 op/s 2023-08-25T02:28:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:55 smithi121 ceph-mon[100099]: pgmap v75: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.4 MiB/s wr, 14 op/s 2023-08-25T02:28:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:57 smithi089 ceph-mon[98732]: pgmap v76: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 20 op/s 2023-08-25T02:28:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:57 smithi121 ceph-mon[100099]: pgmap v76: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 20 op/s 2023-08-25T02:28:57.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:57 smithi122 ceph-mon[100457]: pgmap v76: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 20 op/s 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 -- 172.21.15.89:0/1646442808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e700ffa80 msgr2=0x7f9e700fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 --2- 172.21.15.89:0/1646442808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e700ffa80 0x7f9e700fff00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e600099c0 tx=0x7f9e60009cd0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 -- 172.21.15.89:0/1646442808 shutdown_connections 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 --2- 172.21.15.89:0/1646442808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 0x7f9e70105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 --2- 172.21.15.89:0/1646442808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e70100440 0x7f9e7010c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 --2- 172.21.15.89:0/1646442808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e700ffa80 0x7f9e700fff00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 -- 172.21.15.89:0/1646442808 >> 172.21.15.89:0/1646442808 conn(0x7f9e70075250 msgr2=0x7f9e70075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 -- 172.21.15.89:0/1646442808 shutdown_connections 2023-08-25T02:28:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.879+0000 7f9e779c3700 1 -- 172.21.15.89:0/1646442808 wait complete. 2023-08-25T02:28:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 Processor -- start 2023-08-25T02:28:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 -- start start 2023-08-25T02:28:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e700ffa80 0x7f9e7019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 0x7f9e70196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70197010 con 0x7f9e70100440 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70197180 con 0x7f9e70105840 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e779c3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70197340 con 0x7f9e700ffa80 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e74f5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e7575f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e700ffa80 0x7f9e7019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e74f5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48808/0 (socket says 172.21.15.89:48808) 2023-08-25T02:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e74f5e700 1 -- 172.21.15.89:0/4087358430 learned_addr learned my addr 172.21.15.89:0/4087358430 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.880+0000 7f9e75f60700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 0x7f9e70196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 -- 172.21.15.89:0/4087358430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e700ffa80 msgr2=0x7f9e7019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e700ffa80 0x7f9e7019ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 -- 172.21.15.89:0/4087358430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 msgr2=0x7f9e70196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 0x7f9e70196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 -- 172.21.15.89:0/4087358430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600095f0 con 0x7f9e70100440 2023-08-25T02:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e60000c00 tx=0x7f9e6000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e667fc700 1 -- 172.21.15.89:0/4087358430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6001d070 con 0x7f9e70100440 2023-08-25T02:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e667fc700 1 -- 172.21.15.89:0/4087358430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e60005050 con 0x7f9e70100440 2023-08-25T02:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e701975c0 con 0x7f9e70100440 2023-08-25T02:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e667fc700 1 -- 172.21.15.89:0/4087358430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60004b70 con 0x7f9e70100440 2023-08-25T02:28:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.881+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e701a39f0 con 0x7f9e70100440 2023-08-25T02:28:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.883+0000 7f9e667fc700 1 -- 172.21.15.89:0/4087358430 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f9e60004cd0 con 0x7f9e70100440 2023-08-25T02:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.884+0000 7f9e667fc700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 0x7f9e5c0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.884+0000 7f9e7575f700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 0x7f9e5c0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.884+0000 7f9e667fc700 1 -- 172.21.15.89:0/4087358430 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f9e600cc430 con 0x7f9e70100440 2023-08-25T02:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.884+0000 7f9e7575f700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 0x7f9e5c0a2580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005950 tx=0x7f9e6c00b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 Processor -- start 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 -- start start 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8294320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c828edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c81f6750 con 0x55a9c8294860 2023-08-25T02:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c81e7720 con 0x55a9c8293f40 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c81e5040 con 0x55a9c828e990 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e74f5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c828edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c828edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40754/0 (socket says 172.21.15.89:40754) 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e75f60700 1 -- 172.21.15.89:0/3502308429 learned_addr learned my addr 172.21.15.89:0/3502308429 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.885+0000 7f9e7575f700 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8294320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 -- 172.21.15.89:0/3502308429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 msgr2=0x55a9c828edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c828edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 -- 172.21.15.89:0/3502308429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 msgr2=0x55a9c8294320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8294320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 -- 172.21.15.89:0/3502308429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9c80605d0 con 0x55a9c8294860 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e7575f700 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8294320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:28:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828e450 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e600c9b20 tx=0x7f9e60099550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25151ed886e45d6b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/3502308429 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6009d9a0 con 0x55a9c8294860 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/3502308429 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e600a18f0 con 0x55a9c8294860 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.886+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/3502308429 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6009a060 con 0x55a9c8294860 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- 172.21.15.89:0/3502308429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 msgr2=0x55a9c828e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828e450 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e600c9b20 tx=0x7f9e60099550 comp rx=0 tx=0).stop 2023-08-25T02:28:58.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- 172.21.15.89:0/3502308429 shutdown_connections 2023-08-25T02:28:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8294320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c828edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/3502308429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828e450 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- 172.21.15.89:0/3502308429 >> 172.21.15.89:0/3502308429 conn(0x55a9c8203080 msgr2=0x55a9c8288d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:58.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- 172.21.15.89:0/3502308429 shutdown_connections 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- 172.21.15.89:0/3502308429 wait complete. 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- start start 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 Processor -- start 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8297970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8294860 0x55a9c82980b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.887+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c81e0930 con 0x55a9c828e990 2023-08-25T02:28:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c817eb50 con 0x55a9c8293f40 2023-08-25T02:28:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c817e850 con 0x55a9c8294860 2023-08-25T02:28:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 10 client.0 _ll_register_callbacks cb 0x55a9c80c8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48820/0 (socket says 172.21.15.89:48820) 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 -- 172.21.15.89:0/1356540515 learned_addr learned my addr 172.21.15.89:0/1356540515 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8297970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e75f60700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8294860 0x55a9c82980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 -- 172.21.15.89:0/1356540515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8294860 msgr2=0x55a9c82980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8294860 0x55a9c82980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 -- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 msgr2=0x55a9c8297970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8297970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 -- 172.21.15.89:0/1356540515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e701a39f0 con 0x55a9c828e990 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8297970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e7575f700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c011740 tx=0x7f9e6c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea90c4146232ee2b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c004a30 con 0x55a9c828e990 2023-08-25T02:28:58.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f9e6c017070 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9c825e160 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e6c010eb0 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 10 client.15519 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.888+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a9c800cd80 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a9c825e160 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9e6c01f640 con 0x55a9c828e990 2023-08-25T02:28:58.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 20 client.15519 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82a7e90 0x55a9c82aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 4 client.15519 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9c8186b40 con 0x55a9c82a7e90 2023-08-25T02:28:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82a7e90 0x55a9c82aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e58ff9700 10 client.15519 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.889+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82a7e90 0x55a9c82aa2b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fffcec036be5dc3a server_cookie=7dfdbc24c2a11b2d in_seq=0 out_seq=0 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1356540515 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a9c8186b40 con 0x55a9c82a7e90 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e58ff9700 10 client.15519 handle_command_reply: tid=0 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 client.15519 shutdown 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 msgr2=0x55a9c829a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c011740 tx=0x7f9e6c0047c0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 shutdown_connections 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82a7e90 0x55a9c82aa2b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8294860 0x55a9c82980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e58ff9700 0 client.15519 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c828e990 0x55a9c829a6f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1356540515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8297970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 >> 172.21.15.89:0/1356540515 conn(0x55a9c8203080 msgr2=0x55a9c8283820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 shutdown_connections 2023-08-25T02:28:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.890+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1356540515 wait complete. 2023-08-25T02:28:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.891+0000 7f9e83b67b80 20 client.15519 trim_cache size 0 max 16384 2023-08-25T02:28:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.905+0000 7f9e83b67b80 1 Processor -- start 2023-08-25T02:28:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.905+0000 7f9e83b67b80 1 -- start start 2023-08-25T02:28:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.905+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c828e990 0x55a9c828e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.905+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8293f40 0x55a9c829ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c80628c0 con 0x55a9c8294860 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c8298690 con 0x55a9c828e990 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c8289fb0 con 0x55a9c8293f40 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48832/0 (socket says 172.21.15.89:48832) 2023-08-25T02:28:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 -- 172.21.15.89:0/2762864264 learned_addr learned my addr 172.21.15.89:0/2762864264 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e7575f700 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c828e990 0x55a9c828e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8293f40 0x55a9c829ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 -- 172.21.15.89:0/2762864264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8293f40 msgr2=0x55a9c829ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8293f40 0x55a9c829ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 -- 172.21.15.89:0/2762864264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c828e990 msgr2=0x55a9c828e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c828e990 0x55a9c828e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 -- 172.21.15.89:0/2762864264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9c828f040 con 0x55a9c8294860 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e75f60700 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9e680040c0 tx=0x7f9e68009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff263acd5127ffc2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e5bfff700 1 -- 172.21.15.89:0/2762864264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800e040 con 0x55a9c8294860 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e5bfff700 1 -- 172.21.15.89:0/2762864264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e6800f040 con 0x55a9c8294860 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.906+0000 7f9e5bfff700 1 -- 172.21.15.89:0/2762864264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68005020 con 0x55a9c8294860 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- 172.21.15.89:0/2762864264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 msgr2=0x55a9c8288bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9e680040c0 tx=0x7f9e68009920 comp rx=0 tx=0).stop 2023-08-25T02:28:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- 172.21.15.89:0/2762864264 shutdown_connections 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c828e990 0x55a9c828e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c8288bd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/2762864264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c8293f40 0x55a9c829ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- 172.21.15.89:0/2762864264 >> 172.21.15.89:0/2762864264 conn(0x55a9c8203080 msgr2=0x55a9c82a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- 172.21.15.89:0/2762864264 shutdown_connections 2023-08-25T02:28:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- 172.21.15.89:0/2762864264 wait complete. 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 -- start start 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.907+0000 7f9e83b67b80 1 Processor -- start 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c82909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8290f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c828fe30 con 0x55a9c8294860 2023-08-25T02:28:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c828ffa0 con 0x55a9c8293f40 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9c8290160 con 0x55a9c828e990 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 10 client.0 _ll_register_callbacks cb 0x7f9e6c010cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e7575f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c82909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48838/0 (socket says 172.21.15.89:48838) 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e74f5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8290f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 -- 172.21.15.89:0/1819415598 learned_addr learned my addr 172.21.15.89:0/1819415598 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:28:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 -- 172.21.15.89:0/1819415598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 msgr2=0x55a9c82909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c82909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 -- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 msgr2=0x55a9c8290f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8290f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 -- 172.21.15.89:0/1819415598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e68003cf0 con 0x55a9c8294860 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e74f5e700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8290f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e75f60700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b010 tx=0x7f9e68006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf5062c2aff0c58 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:28:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800e050 con 0x55a9c8294860 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f9e68005020 con 0x55a9c8294860 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.908+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9c8281600 con 0x55a9c8294860 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e680051e0 con 0x55a9c8294860 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 10 client.15525 fetch_fsmap learned FSMap version 28 2023-08-25T02:28:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a9c7ff5c00 con 0x55a9c8294860 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a9c8281600 con 0x55a9c8294860 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9e6802e070 con 0x55a9c8294860 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 20 client.15525 populate_metadata read hostname 'smithi089' 2023-08-25T02:28:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82fcaf0 0x55a9c82fef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:28:58.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 4 client.15525 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:28:58.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a9c81d9910 con 0x55a9c82fcaf0 2023-08-25T02:28:58.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.909+0000 7f9e7575f700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82fcaf0 0x55a9c82fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:28:58.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.910+0000 7f9e58ff9700 10 client.15525 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:58.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:58.910+0000 7f9e7575f700 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82fcaf0 0x55a9c82fef10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77e901fd65610f05 server_cookie=f5ee9f77b2984f17 in_seq=0 out_seq=0 2023-08-25T02:28:59.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e58ff9700 1 -- 172.21.15.89:0/1819415598 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a9c81d9910 con 0x55a9c82fcaf0 2023-08-25T02:28:59.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e58ff9700 10 client.15525 handle_command_reply: tid=0 2023-08-25T02:28:59.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 client.15525 shutdown 2023-08-25T02:28:59.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 msgr2=0x55a9c828f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:59.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b010 tx=0x7f9e68006040 comp rx=0 tx=0).stop 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 shutdown_connections 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55a9c82fcaf0 0x55a9c82fef10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9c8294860 0x55a9c828f8f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e58ff9700 0 client.15525 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a9c828e990 0x55a9c82909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 --2- 172.21.15.89:0/1819415598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a9c8293f40 0x55a9c8290f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.612+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 >> 172.21.15.89:0/1819415598 conn(0x55a9c8203080 msgr2=0x55a9c8288bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:59.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.613+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 shutdown_connections 2023-08-25T02:28:59.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.613+0000 7f9e83b67b80 1 -- 172.21.15.89:0/1819415598 wait complete. 2023-08-25T02:28:59.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.613+0000 7f9e83b67b80 20 client.15525 trim_cache size 0 max 16384 2023-08-25T02:28:59.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:28:59.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:28:59.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:28:59.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:28:59.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 msgr2=0x7f9e5c0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:59.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 0x7f9e5c0a2580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005950 tx=0x7f9e6c00b410 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 msgr2=0x7f9e7019cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9e60000c00 tx=0x7f9e6000f690 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 shutdown_connections 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9e5c0a00c0 0x7f9e5c0a2580 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e70105840 0x7f9e70196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e700ffa80 0x7f9e7019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 --2- 172.21.15.89:0/4087358430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e70100440 0x7f9e7019cfb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:28:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 >> 172.21.15.89:0/4087358430 conn(0x7f9e70075250 msgr2=0x7f9e70067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:28:59.619 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 shutdown_connections 2023-08-25T02:28:59.619 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:28:59.614+0000 7f9e779c3700 1 -- 172.21.15.89:0/4087358430 wait complete. 2023-08-25T02:28:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:28:59 smithi089 ceph-mon[98732]: pgmap v77: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 12 op/s 2023-08-25T02:28:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:28:59 smithi121 ceph-mon[100099]: pgmap v77: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 12 op/s 2023-08-25T02:28:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:28:59 smithi122 ceph-mon[100457]: pgmap v77: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 12 op/s 2023-08-25T02:29:01.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:01 smithi121 ceph-mon[100099]: pgmap v78: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.9 MiB/s wr, 12 op/s 2023-08-25T02:29:01.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:01 smithi122 ceph-mon[100457]: pgmap v78: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.9 MiB/s wr, 12 op/s 2023-08-25T02:29:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:01 smithi089 ceph-mon[98732]: pgmap v78: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.9 MiB/s wr, 12 op/s 2023-08-25T02:29:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:42200000 of 50000000 tuples (84%) done (elapsed 141.60 s, remaining 26.17 s) 2023-08-25T02:29:02.398 INFO:teuthology.orchestra.run.smithi089.stderr:42300000 of 50000000 tuples (84%) done (elapsed 141.70 s, remaining 25.79 s) 2023-08-25T02:29:02.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:02 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:02.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:02 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:02.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:02 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:03.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:03 smithi121 ceph-mon[100099]: pgmap v79: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:03.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:03 smithi122 ceph-mon[100457]: pgmap v79: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:03 smithi089 ceph-mon[98732]: pgmap v79: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:05.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:05 smithi122 ceph-mon[100457]: pgmap v80: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:05 smithi089 ceph-mon[98732]: pgmap v80: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:05 smithi121 ceph-mon[100099]: pgmap v80: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:07.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:07 smithi122 ceph-mon[100457]: pgmap v81: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:07 smithi089 ceph-mon[98732]: pgmap v81: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:07 smithi121 ceph-mon[100099]: pgmap v81: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 34 op/s 2023-08-25T02:29:08.705 INFO:teuthology.orchestra.run.smithi089.stderr:42400000 of 50000000 tuples (84%) done (elapsed 148.00 s, remaining 26.53 s) 2023-08-25T02:29:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:09 smithi122 ceph-mon[100457]: pgmap v82: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:09 smithi089 ceph-mon[98732]: pgmap v82: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:09 smithi121 ceph-mon[100099]: pgmap v82: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:11.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:11 smithi121 ceph-mon[100099]: pgmap v83: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:11.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:11 smithi122 ceph-mon[100457]: pgmap v83: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:11.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:11 smithi089 ceph-mon[98732]: pgmap v83: 97 pgs: 97 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 13 MiB/s wr, 28 op/s 2023-08-25T02:29:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/3450935692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 msgr2=0x7fe3f0105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/3450935692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f0105c20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40099c0 tx=0x7fe3e4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/3450935692 shutdown_connections 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/3450935692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f0100440 0x7fe3f010c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/3450935692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f0105c20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/3450935692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f00ffa80 0x7fe3f00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/3450935692 >> 172.21.15.89:0/3450935692 conn(0x7fe3f0075250 msgr2=0x7fe3f0075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/3450935692 shutdown_connections 2023-08-25T02:29:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.736+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/3450935692 wait complete. 2023-08-25T02:29:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 Processor -- start 2023-08-25T02:29:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 -- start start 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f00ffa80 0x7fe3f0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f0100440 0x7fe3f0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01a2db0 con 0x7fe3f0105840 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01a2f20 con 0x7fe3f00ffa80 2023-08-25T02:29:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f6a6f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01a30e0 con 0x7fe3f0100440 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48852/0 (socket says 172.21.15.89:48852) 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3effff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f0100440 0x7fe3f0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 -- 172.21.15.89:0/4068124351 learned_addr learned my addr 172.21.15.89:0/4068124351 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f480b700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f00ffa80 0x7fe3f0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 -- 172.21.15.89:0/4068124351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f0100440 msgr2=0x7fe3f0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f0100440 0x7fe3f0197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 -- 172.21.15.89:0/4068124351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f00ffa80 msgr2=0x7fe3f0197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.737+0000 7fe3f500c700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f00ffa80 0x7fe3f0197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3f500c700 1 -- 172.21.15.89:0/4068124351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e40095f0 con 0x7fe3f0105840 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3f500c700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8007ee0 tx=0x7fe3e800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3edffb700 1 -- 172.21.15.89:0/4068124351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8017070 con 0x7fe3f0105840 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3edffb700 1 -- 172.21.15.89:0/4068124351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e800f040 con 0x7fe3f0105840 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f01a33c0 con 0x7fe3f0105840 2023-08-25T02:29:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3edffb700 1 -- 172.21.15.89:0/4068124351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8005740 con 0x7fe3f0105840 2023-08-25T02:29:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.738+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f01a3880 con 0x7fe3f0105840 2023-08-25T02:29:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.740+0000 7fe3edffb700 1 -- 172.21.15.89:0/4068124351 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fe3e800dc60 con 0x7fe3f0105840 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.741+0000 7fe3edffb700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 0x7fe3e00a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.741+0000 7fe3edffb700 1 -- 172.21.15.89:0/4068124351 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe3e80cae40 con 0x7fe3f0105840 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.741+0000 7fe3f480b700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 0x7fe3e00a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.741+0000 7fe3f480b700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 0x7fe3e00a2580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4000c00 tx=0x7fe3e4019040 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 Processor -- start 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 -- start start 2023-08-25T02:29:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0f0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0eadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e052750 con 0x55916e0f0860 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e043720 con 0x55916e0ea990 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe402c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e041040 con 0x55916e0eff40 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48858/0 (socket says 172.21.15.89:48858) 2023-08-25T02:29:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 -- 172.21.15.89:0/2307170650 learned_addr learned my addr 172.21.15.89:0/2307170650 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3f480b700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3f500c700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0eadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 -- 172.21.15.89:0/2307170650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 msgr2=0x55916e0f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 -- 172.21.15.89:0/2307170650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 msgr2=0x55916e0eadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0eadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 -- 172.21.15.89:0/2307170650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55916debc5d0 con 0x55916e0f0860 2023-08-25T02:29:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3f500c700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0eadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.742+0000 7fe3effff700 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009d20 tx=0x7fe3dc011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f435a6120b8103d6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe3daffd700 1 -- 172.21.15.89:0/2307170650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc019630 con 0x55916e0f0860 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe3daffd700 1 -- 172.21.15.89:0/2307170650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3dc019c50 con 0x55916e0f0860 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe3daffd700 1 -- 172.21.15.89:0/2307170650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc00eb90 con 0x55916e0f0860 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 -- 172.21.15.89:0/2307170650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 msgr2=0x55916e0ea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009d20 tx=0x7fe3dc011720 comp rx=0 tx=0).stop 2023-08-25T02:29:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 -- 172.21.15.89:0/2307170650 shutdown_connections 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0eadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2307170650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0ea450 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 -- 172.21.15.89:0/2307170650 >> 172.21.15.89:0/2307170650 conn(0x55916e05f080 msgr2=0x55916e0e4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 -- 172.21.15.89:0/2307170650 shutdown_connections 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.743+0000 7fe402c13b80 1 -- 172.21.15.89:0/2307170650 wait complete. 2023-08-25T02:29:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 -- start start 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 Processor -- start 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 0x55916e0f3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e03c930 con 0x55916e0f0860 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916dfdab50 con 0x55916e0eff40 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916dfda850 con 0x55916e0ea990 2023-08-25T02:29:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe402c13b80 10 client.0 _ll_register_callbacks cb 0x55916df24a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe3f480b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe3f480b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40798/0 (socket says 172.21.15.89:40798) 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe3f480b700 1 -- 172.21.15.89:0/2234490442 learned_addr learned my addr 172.21.15.89:0/2234490442 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.744+0000 7fe3f500c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3effff700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 0x55916e0f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 -- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 msgr2=0x55916e0f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 0x55916e0f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 -- 172.21.15.89:0/2234490442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 msgr2=0x55916e0f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0f3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 -- 172.21.15.89:0/2234490442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc009980 con 0x55916e0ea990 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f500c700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3effff700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 0x55916e0f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3f480b700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40099c0 tx=0x7fe3e4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8491a614da18d2f3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3cffff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:29:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4031070 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe3e4005250 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55916e0ba160 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3e401c040 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe402c13b80 10 client.25159 fetch_fsmap learned FSMap version 29 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55916de68d80 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.745+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e4042080 con 0x55916e0ea990 2023-08-25T02:29:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 29) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3e403c030 con 0x55916e0ea990 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 10 client.25159 fetch_fsmap finished waiting for FSMap version 29 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 10 client.25159 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 20 client.25159 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e104250 0x55916e106670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 4 client.25159 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55916dfe2b40 con 0x55916e104250 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe3effff700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e104250 0x55916e106670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe3cffff700 10 client.25159 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.746+0000 7fe3effff700 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e104250 0x55916e106670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2793ff51ac6f2df4 server_cookie=96f7e23dfc6f8652 in_seq=0 out_seq=0 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe3cffff700 1 -- 172.21.15.89:0/2234490442 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55916dfe2b40 con 0x55916e104250 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe3cffff700 10 client.25159 handle_command_reply: tid=0 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 client.25159 shutdown 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 msgr2=0x55916e0f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40099c0 tx=0x7fe3e4009310 comp rx=0 tx=0).stop 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 shutdown_connections 2023-08-25T02:29:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e104250 0x55916e106670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e0f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f6680 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe3cffff700 0 client.25159 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 --2- 172.21.15.89:0/2234490442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0eff40 0x55916e0f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 >> 172.21.15.89:0/2234490442 conn(0x55916e05f080 msgr2=0x55916e0df820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 shutdown_connections 2023-08-25T02:29:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 1 -- 172.21.15.89:0/2234490442 wait complete. 2023-08-25T02:29:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.747+0000 7fe402c13b80 20 client.25159 trim_cache size 0 max 16384 2023-08-25T02:29:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 Processor -- start 2023-08-25T02:29:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 -- start start 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0fb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916debe8c0 con 0x55916e0f0860 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e0f4900 con 0x55916e0ea990 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe402c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e0f4ee0 con 0x55916e0eff40 2023-08-25T02:29:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3effff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0fb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48874/0 (socket says 172.21.15.89:48874) 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 -- 172.21.15.89:0/1794223176 learned_addr learned my addr 172.21.15.89:0/1794223176 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f480b700 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 -- 172.21.15.89:0/1794223176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 msgr2=0x55916e0fb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0fb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 -- 172.21.15.89:0/1794223176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 msgr2=0x55916e0f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3f500c700 1 -- 172.21.15.89:0/1794223176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55916de51c00 con 0x55916e0f0860 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.762+0000 7fe3effff700 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe3f500c700 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8098250 tx=0x7fe3e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dae53aac936171f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe3db7fe700 1 -- 172.21.15.89:0/1794223176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e809c1b0 con 0x55916e0f0860 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe3db7fe700 1 -- 172.21.15.89:0/1794223176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e8098550 con 0x55916e0f0860 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe3db7fe700 1 -- 172.21.15.89:0/1794223176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80987b0 con 0x55916e0f0860 2023-08-25T02:29:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 -- 172.21.15.89:0/1794223176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 msgr2=0x55916e100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8098250 tx=0x7fe3e800b040 comp rx=0 tx=0).stop 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 -- 172.21.15.89:0/1794223176 shutdown_connections 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0ea990 0x55916e0f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0f0860 0x55916e100480 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 --2- 172.21.15.89:0/1794223176 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0eff40 0x55916e0fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 -- 172.21.15.89:0/1794223176 >> 172.21.15.89:0/1794223176 conn(0x55916e05f080 msgr2=0x55916e0e4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 -- 172.21.15.89:0/1794223176 shutdown_connections 2023-08-25T02:29:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.763+0000 7fe402c13b80 1 -- 172.21.15.89:0/1794223176 wait complete. 2023-08-25T02:29:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 -- start start 2023-08-25T02:29:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 Processor -- start 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 0x55916e0f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e157240 con 0x55916e0eff40 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e1573b0 con 0x55916e0f0860 2023-08-25T02:29:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916e157570 con 0x55916e0ea990 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe402c13b80 10 client.0 _ll_register_callbacks cb 0x7fe3e400be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48878/0 (socket says 172.21.15.89:48878) 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 -- 172.21.15.89:0/3703249321 learned_addr learned my addr 172.21.15.89:0/3703249321 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3f480b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3f500c700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 0x55916e0f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 -- 172.21.15.89:0/3703249321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 msgr2=0x55916e0f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 -- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 msgr2=0x55916e0f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 0x55916e0f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3effff700 1 -- 172.21.15.89:0/3703249321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55916debc5d0 con 0x55916e0eff40 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.764+0000 7fe3f500c700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 0x55916e0f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3effff700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc01c040 tx=0x7fe3dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ce192f3a7a637a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc029040 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe3dc02a420 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55916e0dd600 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3dc005040 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 10 client.15543 fetch_fsmap learned FSMap version 29 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fe3e8094040 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55916e0dd600 con 0x55916e0eff40 2023-08-25T02:29:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3dc02ab90 con 0x55916e0eff40 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 10 client.15543 fetch_fsmap finished waiting for FSMap version 29 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 20 client.15543 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e158cd0 0x55916e15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 4 client.15543 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.765+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55916e035910 con 0x55916e158cd0 2023-08-25T02:29:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.766+0000 7fe3f480b700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e158cd0 0x55916e15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.766+0000 7fe3cffff700 10 client.15543 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:12.766+0000 7fe3f480b700 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e158cd0 0x55916e15b0f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b09cb2bd3d9b03 server_cookie=4307d777e387ba65 in_seq=0 out_seq=0 2023-08-25T02:29:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe3cffff700 1 -- 172.21.15.89:0/3703249321 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55916e035910 con 0x55916e158cd0 2023-08-25T02:29:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe3cffff700 10 client.15543 handle_command_reply: tid=0 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 client.15543 shutdown 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 msgr2=0x55916e0f7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc01c040 tx=0x7fe3dc000bc0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 shutdown_connections 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55916e158cd0 0x55916e15b0f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55916e0f0860 0x55916e0f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55916e0ea990 0x55916e0f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 --2- 172.21.15.89:0/3703249321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55916e0eff40 0x55916e0f7760 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 >> 172.21.15.89:0/3703249321 conn(0x55916e05f080 msgr2=0x55916e0edb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe3cffff700 0 client.15543 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 shutdown_connections 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 1 -- 172.21.15.89:0/3703249321 wait complete. 2023-08-25T02:29:13.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.148+0000 7fe402c13b80 20 client.15543 trim_cache size 0 max 16384 2023-08-25T02:29:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:29:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:29:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:29:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 msgr2=0x7fe3e00a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 0x7fe3e00a2580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4000c00 tx=0x7fe3e4019040 comp rx=0 tx=0).stop 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 msgr2=0x7fe3f01a27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8007ee0 tx=0x7fe3e800d3b0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 shutdown_connections 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe3e00a00c0 0x7fe3e00a2580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3f00ffa80 0x7fe3f0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3f0105840 0x7fe3f01a27e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 --2- 172.21.15.89:0/4068124351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3f0100440 0x7fe3f0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 >> 172.21.15.89:0/4068124351 conn(0x7fe3f0075250 msgr2=0x7fe3f00fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 shutdown_connections 2023-08-25T02:29:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:13.149+0000 7fe3f6a6f700 1 -- 172.21.15.89:0/4068124351 wait complete. 2023-08-25T02:29:13.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:13 smithi122 ceph-mon[100457]: pgmap v84: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 33 op/s 2023-08-25T02:29:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:13 smithi089 ceph-mon[98732]: pgmap v84: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 33 op/s 2023-08-25T02:29:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:13 smithi121 ceph-mon[100099]: pgmap v84: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 33 op/s 2023-08-25T02:29:13.974 INFO:teuthology.orchestra.run.smithi089.stderr:42500000 of 50000000 tuples (85%) done (elapsed 153.27 s, remaining 27.05 s) 2023-08-25T02:29:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:42600000 of 50000000 tuples (85%) done (elapsed 153.37 s, remaining 26.64 s) 2023-08-25T02:29:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:15 smithi122 ceph-mon[100457]: pgmap v85: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 MiB/s wr, 11 op/s 2023-08-25T02:29:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:15 smithi089 ceph-mon[98732]: pgmap v85: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 MiB/s wr, 11 op/s 2023-08-25T02:29:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:15 smithi121 ceph-mon[100099]: pgmap v85: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 MiB/s wr, 11 op/s 2023-08-25T02:29:17.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:17 smithi122 ceph-mon[100457]: pgmap v86: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 19 op/s 2023-08-25T02:29:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:17 smithi089 ceph-mon[98732]: pgmap v86: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 19 op/s 2023-08-25T02:29:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:17 smithi121 ceph-mon[100099]: pgmap v86: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 MiB/s wr, 19 op/s 2023-08-25T02:29:19.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:19 smithi089 ceph-mon[98732]: pgmap v87: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 12 op/s 2023-08-25T02:29:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:19 smithi121 ceph-mon[100099]: pgmap v87: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 12 op/s 2023-08-25T02:29:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:19 smithi122 ceph-mon[100457]: pgmap v87: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 12 op/s 2023-08-25T02:29:21.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:21 smithi121 ceph-mon[100099]: pgmap v88: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 12 op/s 2023-08-25T02:29:21.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:21 smithi122 ceph-mon[100457]: pgmap v88: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 12 op/s 2023-08-25T02:29:21.518 INFO:teuthology.orchestra.run.smithi089.stderr:42700000 of 50000000 tuples (85%) done (elapsed 160.82 s, remaining 27.49 s) 2023-08-25T02:29:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:21 smithi089 ceph-mon[98732]: pgmap v88: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 12 op/s 2023-08-25T02:29:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:23 smithi122 ceph-mon[100457]: pgmap v89: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 17 op/s 2023-08-25T02:29:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:23 smithi089 ceph-mon[98732]: pgmap v89: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 17 op/s 2023-08-25T02:29:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:23 smithi121 ceph-mon[100099]: pgmap v89: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.4 MiB/s wr, 17 op/s 2023-08-25T02:29:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:25 smithi122 ceph-mon[100457]: pgmap v90: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2023-08-25T02:29:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:25 smithi089 ceph-mon[98732]: pgmap v90: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2023-08-25T02:29:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:25 smithi121 ceph-mon[100099]: pgmap v90: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 12 op/s 2023-08-25T02:29:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.269+0000 7ff830a99700 1 -- 172.21.15.89:0/301196620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c100440 msgr2=0x7ff82c10c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.269+0000 7ff830a99700 1 --2- 172.21.15.89:0/301196620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c100440 0x7ff82c10c650 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff824009fc0 tx=0x7ff82400b330 comp rx=0 tx=0).stop 2023-08-25T02:29:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 -- 172.21.15.89:0/301196620 shutdown_connections 2023-08-25T02:29:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 --2- 172.21.15.89:0/301196620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c100440 0x7ff82c10c650 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 --2- 172.21.15.89:0/301196620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c105840 0x7ff82c105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 --2- 172.21.15.89:0/301196620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c0ffa80 0x7ff82c0fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 -- 172.21.15.89:0/301196620 >> 172.21.15.89:0/301196620 conn(0x7ff82c075250 msgr2=0x7ff82c075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 -- 172.21.15.89:0/301196620 shutdown_connections 2023-08-25T02:29:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.270+0000 7ff830a99700 1 -- 172.21.15.89:0/301196620 wait complete. 2023-08-25T02:29:26.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 Processor -- start 2023-08-25T02:29:26.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 -- start start 2023-08-25T02:29:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 0x7ff82c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c105840 0x7ff82c1a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1a2ea0 con 0x7ff82c0ffa80 2023-08-25T02:29:26.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1a3010 con 0x7ff82c105840 2023-08-25T02:29:26.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff830a99700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1a31d0 con 0x7ff82c100440 2023-08-25T02:29:26.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff82a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 0x7ff82c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40826/0 (socket says 172.21.15.89:40826) 2023-08-25T02:29:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff829d9b700 1 -- 172.21.15.89:0/1642986395 learned_addr learned my addr 172.21.15.89:0/1642986395 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.271+0000 7ff82ad9d700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c105840 0x7ff82c1a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 -- 172.21.15.89:0/1642986395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c105840 msgr2=0x7ff82c1a28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c105840 0x7ff82c1a28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 -- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 msgr2=0x7ff82c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 0x7ff82c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 -- 172.21.15.89:0/1642986395 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8200095f0 con 0x7ff82c100440 2023-08-25T02:29:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff82a59c700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 0x7ff82c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:26.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff829d9b700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff82000ec10 tx=0x7ff82000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.283 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff81b7fe700 1 -- 172.21.15.89:0/1642986395 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82000cc90 con 0x7ff82c100440 2023-08-25T02:29:26.283 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff824009c40 con 0x7ff82c100440 2023-08-25T02:29:26.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff81b7fe700 1 -- 172.21.15.89:0/1642986395 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8200044e0 con 0x7ff82c100440 2023-08-25T02:29:26.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff81b7fe700 1 -- 172.21.15.89:0/1642986395 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820010430 con 0x7ff82c100440 2023-08-25T02:29:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.272+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c1a3830 con 0x7ff82c100440 2023-08-25T02:29:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.274+0000 7ff81b7fe700 1 -- 172.21.15.89:0/1642986395 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7ff820003fb0 con 0x7ff82c100440 2023-08-25T02:29:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.275+0000 7ff81b7fe700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 0x7ff8140a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.275+0000 7ff82a59c700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 0x7ff8140a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.275+0000 7ff81b7fe700 1 -- 172.21.15.89:0/1642986395 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7ff820014070 con 0x7ff82c100440 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff82a59c700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 0x7ff8140a2580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff81c011490 tx=0x7ff81c009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 Processor -- start 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 -- start start 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0111320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb010bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0073750 con 0x55fdb010b990 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0064720 con 0x55fdb0111860 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.276+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0062040 con 0x55fdb0110f40 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff82a59c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0111320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff82ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb010bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:39308/0 (socket says 172.21.15.89:39308) 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 -- 172.21.15.89:0/3657732789 learned_addr learned my addr 172.21.15.89:0/3657732789 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff82a59c700 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0111320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40832/0 (socket says 172.21.15.89:40832) 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 -- 172.21.15.89:0/3657732789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 msgr2=0x55fdb0111320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0111320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 -- 172.21.15.89:0/3657732789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 msgr2=0x55fdb010bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb010bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 -- 172.21.15.89:0/3657732789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdafedd5d0 con 0x55fdb0111860 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff82ad9d700 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff829d9b700 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff820005d70 tx=0x7ff820095720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbcac85d0d284fb3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.277+0000 7ff818ff9700 1 -- 172.21.15.89:0/3657732789 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8200999b0 con 0x55fdb0111860 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff818ff9700 1 -- 172.21.15.89:0/3657732789 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8200992b0 con 0x55fdb0111860 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff818ff9700 1 -- 172.21.15.89:0/3657732789 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8200959c0 con 0x55fdb0111860 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 -- 172.21.15.89:0/3657732789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 msgr2=0x55fdb010b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff820005d70 tx=0x7ff820095720 comp rx=0 tx=0).stop 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 -- 172.21.15.89:0/3657732789 shutdown_connections 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0111320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/3657732789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010b450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.278+0000 7ff83cc3db80 1 -- 172.21.15.89:0/3657732789 >> 172.21.15.89:0/3657732789 conn(0x55fdb0080080 msgr2=0x55fdb0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- 172.21.15.89:0/3657732789 shutdown_connections 2023-08-25T02:29:26.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- 172.21.15.89:0/3657732789 wait complete. 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- start start 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 Processor -- start 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb010b990 0x55fdb01176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0114970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb005d930 con 0x55fdb0111860 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdafffbb50 con 0x55fdb010b990 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdafffb850 con 0x55fdb0110f40 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff83cc3db80 10 client.0 _ll_register_callbacks cb 0x55fdaff45a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff82ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff82a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb010b990 0x55fdb01176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff82ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48906/0 (socket says 172.21.15.89:48906) 2023-08-25T02:29:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff82ad9d700 1 -- 172.21.15.89:0/2580422056 learned_addr learned my addr 172.21.15.89:0/2580422056 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.279+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0114970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 -- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb010b990 msgr2=0x55fdb01176f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb010b990 0x55fdb01176f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 -- 172.21.15.89:0/2580422056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 msgr2=0x55fdb01150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 -- 172.21.15.89:0/2580422056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c10cee0 con 0x55fdb0110f40 2023-08-25T02:29:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff82ad9d700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0114970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff820099240 tx=0x7ff820016490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c739016dc24d114 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820009a10 con 0x55fdb0110f40 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7ff8200a10c0 con 0x55fdb0110f40 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fdb00db160 con 0x55fdb0110f40 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8200992b0 con 0x55fdb0110f40 2023-08-25T02:29:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff83cc3db80 10 client.25177 fetch_fsmap learned FSMap version 29 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ff82c1a3830 con 0x55fdb0110f40 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.280+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fdb00db160 con 0x55fdb0110f40 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 29) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff8200987a0 con 0x55fdb0110f40 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 10 client.25177 fetch_fsmap finished waiting for FSMap version 29 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 10 client.25177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 20 client.25177 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb0124ec0 0x55fdb01272e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 4 client.25177 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fdb0003b40 con 0x55fdb0124ec0 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff82a59c700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb0124ec0 0x55fdb01272e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff811ffb700 10 client.25177 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.281+0000 7ff82a59c700 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb0124ec0 0x55fdb01272e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f00c2c9a9e5bbc86 server_cookie=a0ebd94e34626ccd in_seq=0 out_seq=0 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff811ffb700 1 -- 172.21.15.89:0/2580422056 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fdb0003b40 con 0x55fdb0124ec0 2023-08-25T02:29:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff811ffb700 10 client.25177 handle_command_reply: tid=0 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 client.25177 shutdown 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 msgr2=0x55fdb0114970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0114970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff820099240 tx=0x7ff820016490 comp rx=0 tx=0).stop 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 shutdown_connections 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb0124ec0 0x55fdb01272e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb010b990 0x55fdb01176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff811ffb700 0 client.25177 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb01150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2580422056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0114970 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 >> 172.21.15.89:0/2580422056 conn(0x55fdb0080080 msgr2=0x55fdb0100820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 shutdown_connections 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2580422056 wait complete. 2023-08-25T02:29:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.282+0000 7ff83cc3db80 20 client.25177 trim_cache size 0 max 16384 2023-08-25T02:29:26.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 Processor -- start 2023-08-25T02:29:26.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 -- start start 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb011c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdafedf8c0 con 0x55fdb010b990 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0102730 con 0x55fdb0111860 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb010b4a0 con 0x55fdb0110f40 2023-08-25T02:29:26.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff82a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb011c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40848/0 (socket says 172.21.15.89:40848) 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff82a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb011c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48910/0 (socket says 172.21.15.89:48910) 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff82ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.298+0000 7ff829d9b700 1 -- 172.21.15.89:0/2148833958 learned_addr learned my addr 172.21.15.89:0/2148833958 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 -- 172.21.15.89:0/2148833958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 msgr2=0x55fdb010eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 -- 172.21.15.89:0/2148833958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 msgr2=0x55fdb011c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb011c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 -- 172.21.15.89:0/2148833958 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdafe89d80 con 0x55fdb0110f40 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff829d9b700 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff820095d20 tx=0x7ff820095720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d9ec95d50f7d6d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff8197fa700 1 -- 172.21.15.89:0/2148833958 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820009a10 con 0x55fdb0110f40 2023-08-25T02:29:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff8197fa700 1 -- 172.21.15.89:0/2148833958 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8200992b0 con 0x55fdb0110f40 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.299+0000 7ff8197fa700 1 -- 172.21.15.89:0/2148833958 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820099510 con 0x55fdb0110f40 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2148833958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 msgr2=0x55fdb0117320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff820095d20 tx=0x7ff820095720 comp rx=0 tx=0).stop 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2148833958 shutdown_connections 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0111860 0x55fdb010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb010b990 0x55fdb011c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/2148833958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb0110f40 0x55fdb0117320 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2148833958 >> 172.21.15.89:0/2148833958 conn(0x55fdb0080080 msgr2=0x55fdb0105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2148833958 shutdown_connections 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- 172.21.15.89:0/2148833958 wait complete. 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 -- start start 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.300+0000 7ff83cc3db80 1 Processor -- start 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 0x55fdb0112ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb0113950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0113e90 con 0x55fdb0111860 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb0114000 con 0x55fdb0110f40 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb01141c0 con 0x55fdb010b990 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 10 client.0 _ll_register_callbacks cb 0x7ff8200091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:39328/0 (socket says 172.21.15.89:39328) 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 -- 172.21.15.89:0/4056556335 learned_addr learned my addr 172.21.15.89:0/4056556335 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff82a59c700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 0x55fdb0112ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff82ad9d700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb0113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 -- 172.21.15.89:0/4056556335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 msgr2=0x55fdb0112ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 0x55fdb0112ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 -- 172.21.15.89:0/4056556335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 msgr2=0x55fdb0113950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb0113950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 -- 172.21.15.89:0/4056556335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c1a3830 con 0x55fdb0110f40 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff82a59c700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 0x55fdb0112ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff829d9b700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff82000c5b0 tx=0x7ff820003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec08d5de4002306b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff811ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820009a10 con 0x55fdb0110f40 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fdb00fe600 con 0x55fdb0110f40 2023-08-25T02:29:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.301+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7ff8200999b0 con 0x55fdb0110f40 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff820098df0 con 0x55fdb0110f40 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 10 client.25163 fetch_fsmap learned FSMap version 29 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fdafe72c00 con 0x55fdb0110f40 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fdb00fe600 con 0x55fdb0110f40 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 29) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff820082ab0 con 0x55fdb0110f40 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 10 client.25163 fetch_fsmap finished waiting for FSMap version 29 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 10 client.25163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 20 client.25163 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb01791d0 0x55fdb017b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 4 client.25163 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fdb0056910 con 0x55fdb01791d0 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.302+0000 7ff82a59c700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb01791d0 0x55fdb017b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.303+0000 7ff811ffb700 10 client.25163 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:26.303+0000 7ff82a59c700 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb01791d0 0x55fdb017b5c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce514b953d307c30 server_cookie=71241bfbaec3857f in_seq=0 out_seq=0 2023-08-25T02:29:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:42800000 of 50000000 tuples (85%) done (elapsed 165.69 s, remaining 27.87 s) 2023-08-25T02:29:26.909 INFO:teuthology.orchestra.run.smithi089.stderr:42900000 of 50000000 tuples (85%) done (elapsed 166.21 s, remaining 27.51 s) 2023-08-25T02:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff811ffb700 1 -- 172.21.15.89:0/4056556335 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fdb0056910 con 0x55fdb01791d0 2023-08-25T02:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff811ffb700 10 client.25163 handle_command_reply: tid=0 2023-08-25T02:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 client.25163 shutdown 2023-08-25T02:29:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 msgr2=0x55fdb0113410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff82000c5b0 tx=0x7ff820003a10 comp rx=0 tx=0).stop 2023-08-25T02:29:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 shutdown_connections 2023-08-25T02:29:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55fdb01791d0 0x55fdb017b5c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fdb0111860 0x55fdb0113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fdb010b990 0x55fdb0112ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff811ffb700 0 client.25163 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 --2- 172.21.15.89:0/4056556335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fdb0110f40 0x55fdb0113410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 >> 172.21.15.89:0/4056556335 conn(0x55fdb0080080 msgr2=0x55fdb01214f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 shutdown_connections 2023-08-25T02:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 1 -- 172.21.15.89:0/4056556335 wait complete. 2023-08-25T02:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.037+0000 7ff83cc3db80 20 client.25163 trim_cache size 0 max 16384 2023-08-25T02:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:29:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 msgr2=0x7ff8140a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 0x7ff8140a2580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff81c011490 tx=0x7ff81c009260 comp rx=0 tx=0).stop 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 msgr2=0x7ff82c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff82000ec10 tx=0x7ff82000ef20 comp rx=0 tx=0).stop 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 shutdown_connections 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ff8140a00c0 0x7ff8140a2580 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff82c105840 0x7ff82c1a28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0ffa80 0x7ff82c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 --2- 172.21.15.89:0/1642986395 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff82c100440 0x7ff82c197f40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 >> 172.21.15.89:0/1642986395 conn(0x7ff82c075250 msgr2=0x7ff82c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 shutdown_connections 2023-08-25T02:29:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:27.038+0000 7ff830a99700 1 -- 172.21.15.89:0/1642986395 wait complete. 2023-08-25T02:29:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:27 smithi089 ceph-mon[98732]: pgmap v91: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.8 MiB/s wr, 26 op/s 2023-08-25T02:29:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:27 smithi121 ceph-mon[100099]: pgmap v91: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.8 MiB/s wr, 26 op/s 2023-08-25T02:29:27.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:27 smithi122 ceph-mon[100457]: pgmap v91: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.8 MiB/s wr, 26 op/s 2023-08-25T02:29:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:29 smithi089 ceph-mon[98732]: pgmap v92: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 19 op/s 2023-08-25T02:29:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:29 smithi121 ceph-mon[100099]: pgmap v92: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 19 op/s 2023-08-25T02:29:29.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:29 smithi122 ceph-mon[100457]: pgmap v92: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 19 op/s 2023-08-25T02:29:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:31 smithi121 ceph-mon[100099]: pgmap v93: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2023-08-25T02:29:31.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:31 smithi122 ceph-mon[100457]: pgmap v93: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2023-08-25T02:29:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:31 smithi089 ceph-mon[98732]: pgmap v93: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 MiB/s wr, 19 op/s 2023-08-25T02:29:31.967 INFO:teuthology.orchestra.run.smithi089.stderr:43000000 of 50000000 tuples (86%) done (elapsed 171.27 s, remaining 27.88 s) 2023-08-25T02:29:32.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:32 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:32.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:32 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:32 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:33.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:33 smithi121 ceph-mon[100099]: pgmap v94: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2023-08-25T02:29:33.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:33 smithi122 ceph-mon[100457]: pgmap v94: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2023-08-25T02:29:33.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:33 smithi089 ceph-mon[98732]: pgmap v94: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2023-08-25T02:29:35.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:35 smithi122 ceph-mon[100457]: pgmap v95: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 52 op/s 2023-08-25T02:29:35.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:35 smithi089 ceph-mon[98732]: pgmap v95: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 52 op/s 2023-08-25T02:29:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:35 smithi121 ceph-mon[100099]: pgmap v95: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.4 MiB/s wr, 52 op/s 2023-08-25T02:29:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:37 smithi089 ceph-mon[98732]: pgmap v96: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 57 op/s 2023-08-25T02:29:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:37 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:29:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:37 smithi121 ceph-mon[100099]: pgmap v96: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 57 op/s 2023-08-25T02:29:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:37 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:29:37.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:37 smithi122 ceph-mon[100457]: pgmap v96: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 MiB/s wr, 57 op/s 2023-08-25T02:29:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:37 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:29:38.448 INFO:teuthology.orchestra.run.smithi089.stderr:43100000 of 50000000 tuples (86%) done (elapsed 177.75 s, remaining 28.46 s) 2023-08-25T02:29:38.551 INFO:teuthology.orchestra.run.smithi089.stderr:43200000 of 50000000 tuples (86%) done (elapsed 177.85 s, remaining 27.99 s) 2023-08-25T02:29:38.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:29:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:29:38.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:29:38.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:29:38.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:38.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:29:38.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:29:38.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:29:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:39 smithi089 ceph-mon[98732]: pgmap v97: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:29:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:39 smithi121 ceph-mon[100099]: pgmap v97: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:29:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:39 smithi122 ceph-mon[100457]: pgmap v97: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.9 MiB/s wr, 44 op/s 2023-08-25T02:29:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 -- 172.21.15.89:0/3468358453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 msgr2=0x7f4e78105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 --2- 172.21.15.89:0/3468358453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e78105c20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0099c0 tx=0x7f4e6c009cd0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 -- 172.21.15.89:0/3468358453 shutdown_connections 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 --2- 172.21.15.89:0/3468358453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 0x7f4e7810c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 --2- 172.21.15.89:0/3468358453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e78105c20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 --2- 172.21.15.89:0/3468358453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 0x7f4e780fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 -- 172.21.15.89:0/3468358453 >> 172.21.15.89:0/3468358453 conn(0x7f4e78075250 msgr2=0x7f4e78075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 -- 172.21.15.89:0/3468358453 shutdown_connections 2023-08-25T02:29:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.156+0000 7f4e7f047700 1 -- 172.21.15.89:0/3468358453 wait complete. 2023-08-25T02:29:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.157+0000 7f4e7f047700 1 Processor -- start 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.157+0000 7f4e7f047700 1 -- start start 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 0x7f4e78197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 0x7f4e78197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781a2ea0 con 0x7f4e78105840 2023-08-25T02:29:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781a3010 con 0x7f4e780ffa80 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7f047700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781a31d0 con 0x7f4e78100440 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 0x7f4e78197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48936/0 (socket says 172.21.15.89:48936) 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 -- 172.21.15.89:0/739831142 learned_addr learned my addr 172.21.15.89:0/739831142 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 0x7f4e78197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 -- 172.21.15.89:0/739831142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 msgr2=0x7f4e78197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 0x7f4e78197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 -- 172.21.15.89:0/739831142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 msgr2=0x7f4e78197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 0x7f4e78197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 -- 172.21.15.89:0/739831142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0095f0 con 0x7f4e78105840 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.158+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000ef30 tx=0x7f4e7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.159+0000 7f4e75ffb700 1 -- 172.21.15.89:0/739831142 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70018070 con 0x7f4e78105840 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.159+0000 7f4e75ffb700 1 -- 172.21.15.89:0/739831142 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e7000f040 con 0x7f4e78105840 2023-08-25T02:29:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.159+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e781a3450 con 0x7f4e78105840 2023-08-25T02:29:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.159+0000 7f4e75ffb700 1 -- 172.21.15.89:0/739831142 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000cd00 con 0x7f4e78105840 2023-08-25T02:29:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.159+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e781a3910 con 0x7f4e78105840 2023-08-25T02:29:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.161+0000 7f4e75ffb700 1 -- 172.21.15.89:0/739831142 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f4e70003a00 con 0x7f4e78105840 2023-08-25T02:29:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.162+0000 7f4e75ffb700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 0x7f4e680a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.162+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 0x7f4e680a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.162+0000 7f4e75ffb700 1 -- 172.21.15.89:0/739831142 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f4e700cb130 con 0x7f4e78105840 2023-08-25T02:29:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.162+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 0x7f4e680a2580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009990 tx=0x7f4e6c005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 Processor -- start 2023-08-25T02:29:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 -- start start 2023-08-25T02:29:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c44e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c49850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1ba8750 con 0x5555b1c45370 2023-08-25T02:29:40.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1b99720 con 0x5555b1c493f0 2023-08-25T02:29:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1b97040 con 0x5555b1c44a50 2023-08-25T02:29:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e7cde3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c44e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48942/0 (socket says 172.21.15.89:48942) 2023-08-25T02:29:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.163+0000 7f4e77fff700 1 -- 172.21.15.89:0/1593090907 learned_addr learned my addr 172.21.15.89:0/1593090907 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c49850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 -- 172.21.15.89:0/1593090907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 msgr2=0x5555b1c44e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c44e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 -- 172.21.15.89:0/1593090907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 msgr2=0x5555b1c49850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c49850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 -- 172.21.15.89:0/1593090907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555b1a125d0 con 0x5555b1c45370 2023-08-25T02:29:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e77fff700 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400bcf0 tx=0x7f4e640103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8981284ad733a94 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e62ffd700 1 -- 172.21.15.89:0/1593090907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64010d10 con 0x5555b1c45370 2023-08-25T02:29:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e62ffd700 1 -- 172.21.15.89:0/1593090907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e64015070 con 0x5555b1c45370 2023-08-25T02:29:40.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.164+0000 7f4e62ffd700 1 -- 172.21.15.89:0/1593090907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64005020 con 0x5555b1c45370 2023-08-25T02:29:40.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/1593090907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 msgr2=0x5555b1c48eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400bcf0 tx=0x7f4e640103d0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/1593090907 shutdown_connections 2023-08-25T02:29:40.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c49850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c44e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/1593090907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c48eb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/1593090907 >> 172.21.15.89:0/1593090907 conn(0x5555b1bb5080 msgr2=0x5555b1c3ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/1593090907 shutdown_connections 2023-08-25T02:29:40.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/1593090907 wait complete. 2023-08-25T02:29:40.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 -- start start 2023-08-25T02:29:40.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.165+0000 7f4e8b1ebb80 1 Processor -- start 2023-08-25T02:29:40.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c50c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c4e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1b92930 con 0x5555b1c45370 2023-08-25T02:29:40.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1b30b50 con 0x5555b1c493f0 2023-08-25T02:29:40.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1b30850 con 0x5555b1c44a50 2023-08-25T02:29:40.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e8b1ebb80 10 client.0 _ll_register_callbacks cb 0x5555b1a7aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:40.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e7cde3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c50c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48948/0 (socket says 172.21.15.89:48948) 2023-08-25T02:29:40.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 -- 172.21.15.89:0/3871691296 learned_addr learned my addr 172.21.15.89:0/3871691296 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c4e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 -- 172.21.15.89:0/3871691296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 msgr2=0x5555b1c50c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c50c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 -- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 msgr2=0x5555b1c4e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c4e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 -- 172.21.15.89:0/3871691296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6400b920 con 0x5555b1c45370 2023-08-25T02:29:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e77fff700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4e64004bc0 tx=0x7f4e64010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424601487ae5c309 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e57fff700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:29:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.166+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64004450 con 0x5555b1c45370 2023-08-25T02:29:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f4e64017070 con 0x5555b1c45370 2023-08-25T02:29:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5555b1c10160 con 0x5555b1c45370 2023-08-25T02:29:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e640045b0 con 0x5555b1c45370 2023-08-25T02:29:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e8b1ebb80 10 client.15573 fetch_fsmap learned FSMap version 30 2023-08-25T02:29:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5555b19bed80 con 0x5555b1c45370 2023-08-25T02:29:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.167+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5555b1c10160 con 0x5555b1c45370 2023-08-25T02:29:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 30) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4e640256a0 con 0x5555b1c45370 2023-08-25T02:29:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 10 client.15573 fetch_fsmap finished waiting for FSMap version 30 2023-08-25T02:29:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 20 client.15573 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1c60980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 4 client.15573 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:29:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5555b1b38b40 con 0x5555b1c5e560 2023-08-25T02:29:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1c60980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e57fff700 10 client.15573 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:40.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.168+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1c60980 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b50b1266ffbaf5d server_cookie=279d1b50d868183 in_seq=0 out_seq=0 2023-08-25T02:29:40.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e57fff700 1 -- 172.21.15.89:0/3871691296 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5555b1b38b40 con 0x5555b1c5e560 2023-08-25T02:29:40.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e57fff700 10 client.15573 handle_command_reply: tid=0 2023-08-25T02:29:40.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 client.15573 shutdown 2023-08-25T02:29:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 msgr2=0x5555b1c4de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4e64004bc0 tx=0x7f4e64010dc0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 shutdown_connections 2023-08-25T02:29:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1c60980 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c493f0 0x5555b1c4e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c44a50 0x5555b1c50c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e57fff700 0 client.15573 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:40.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/3871691296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c4de10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 >> 172.21.15.89:0/3871691296 conn(0x5555b1bb5080 msgr2=0x5555b1c35820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 shutdown_connections 2023-08-25T02:29:40.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/3871691296 wait complete. 2023-08-25T02:29:40.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.169+0000 7f4e8b1ebb80 20 client.15573 trim_cache size 0 max 16384 2023-08-25T02:29:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 Processor -- start 2023-08-25T02:29:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 -- start start 2023-08-25T02:29:40.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c44a50 0x5555b1c50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1c47400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1a148c0 con 0x5555b1c45370 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1c10160 con 0x5555b1c44a50 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1c37720 con 0x5555b1c493f0 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48956/0 (socket says 172.21.15.89:48956) 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 -- 172.21.15.89:0/2333288005 learned_addr learned my addr 172.21.15.89:0/2333288005 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1c47400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c44a50 0x5555b1c50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 -- 172.21.15.89:0/2333288005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 msgr2=0x5555b1c47400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1c47400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 -- 172.21.15.89:0/2333288005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c44a50 msgr2=0x5555b1c50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c44a50 0x5555b1c50730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.184+0000 7f4e77fff700 1 -- 172.21.15.89:0/2333288005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555b19a7c00 con 0x5555b1c45370 2023-08-25T02:29:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e77fff700 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4e64000c00 tx=0x7f4e640103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00c50061084e518 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e637fe700 1 -- 172.21.15.89:0/2333288005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64025440 con 0x5555b1c45370 2023-08-25T02:29:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e637fe700 1 -- 172.21.15.89:0/2333288005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e64015070 con 0x5555b1c45370 2023-08-25T02:29:40.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e637fe700 1 -- 172.21.15.89:0/2333288005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e640038e0 con 0x5555b1c45370 2023-08-25T02:29:40.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/2333288005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 msgr2=0x5555b1c55720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4e64000c00 tx=0x7f4e640103d0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/2333288005 shutdown_connections 2023-08-25T02:29:40.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c44a50 0x5555b1c50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1c47400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/2333288005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c45370 0x5555b1c55720 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/2333288005 >> 172.21.15.89:0/2333288005 conn(0x5555b1bb5080 msgr2=0x5555b1c4b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/2333288005 shutdown_connections 2023-08-25T02:29:40.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.185+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/2333288005 wait complete. 2023-08-25T02:29:40.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 -- start start 2023-08-25T02:29:40.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 Processor -- start 2023-08-25T02:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c45370 0x5555b1c51c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1cb1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1c52550 con 0x5555b1c44a50 2023-08-25T02:29:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1cb18c0 con 0x5555b1c45370 2023-08-25T02:29:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555b1cb1a00 con 0x5555b1c493f0 2023-08-25T02:29:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e8b1ebb80 10 client.0 _ll_register_callbacks cb 0x7f4e6c0037d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7cde3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7cde3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48960/0 (socket says 172.21.15.89:48960) 2023-08-25T02:29:40.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7cde3700 1 -- 172.21.15.89:0/554478927 learned_addr learned my addr 172.21.15.89:0/554478927 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:40.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7d5e4700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1cb1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e77fff700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c45370 0x5555b1c51c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7cde3700 1 -- 172.21.15.89:0/554478927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 msgr2=0x5555b1cb1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.186+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1cb1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e7cde3700 1 -- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c45370 msgr2=0x5555b1c51c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c45370 0x5555b1c51c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e7cde3700 1 -- 172.21.15.89:0/554478927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555b19bed80 con 0x5555b1c44a50 2023-08-25T02:29:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e7cde3700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c000c00 tx=0x7f4e6c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdd935350726a00 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:29:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c02c040 con 0x5555b1c44a50 2023-08-25T02:29:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f4e6c033540 con 0x5555b1c44a50 2023-08-25T02:29:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5555b1c33600 con 0x5555b1c44a50 2023-08-25T02:29:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e6c033980 con 0x5555b1c44a50 2023-08-25T02:29:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 10 client.15579 fetch_fsmap learned FSMap version 30 2023-08-25T02:29:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5555b19e51f0 con 0x5555b1c44a50 2023-08-25T02:29:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5555b1c33600 con 0x5555b1c44a50 2023-08-25T02:29:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 30) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4e6c004650 con 0x5555b1c44a50 2023-08-25T02:29:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 10 client.15579 fetch_fsmap finished waiting for FSMap version 30 2023-08-25T02:29:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 20 client.15579 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1cb5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 4 client.15579 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:29:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.187+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5555b1b8b910 con 0x5555b1c5e560 2023-08-25T02:29:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.188+0000 7f4e77fff700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1cb5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.188+0000 7f4e57fff700 10 client.15579 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.188+0000 7f4e77fff700 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1cb5190 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=604b7639f3514423 server_cookie=6cc61f9fd375bc12 in_seq=0 out_seq=0 2023-08-25T02:29:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e57fff700 1 -- 172.21.15.89:0/554478927 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5555b1b8b910 con 0x5555b1c5e560 2023-08-25T02:29:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e57fff700 10 client.15579 handle_command_reply: tid=0 2023-08-25T02:29:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 client.15579 shutdown 2023-08-25T02:29:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 msgr2=0x5555b1c51700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c000c00 tx=0x7f4e6c009310 comp rx=0 tx=0).stop 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 shutdown_connections 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5555b1c5e560 0x5555b1cb5190 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555b1c493f0 0x5555b1cb1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e57fff700 0 client.15579 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555b1c44a50 0x5555b1c51700 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 --2- 172.21.15.89:0/554478927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5555b1c45370 0x5555b1c51c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.550+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 >> 172.21.15.89:0/554478927 conn(0x5555b1bb5080 msgr2=0x5555b1c4b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 shutdown_connections 2023-08-25T02:29:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e8b1ebb80 1 -- 172.21.15.89:0/554478927 wait complete. 2023-08-25T02:29:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e8b1ebb80 20 client.15579 trim_cache size 0 max 16384 2023-08-25T02:29:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:29:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:29:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:29:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 msgr2=0x7f4e680a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 0x7f4e680a2580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009990 tx=0x7f4e6c005fb0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 msgr2=0x7f4e781a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000ef30 tx=0x7f4e7000c3a0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 shutdown_connections 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4e680a00c0 0x7f4e680a2580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e780ffa80 0x7f4e78197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e78105840 0x7f4e781a28a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 --2- 172.21.15.89:0/739831142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e78100440 0x7f4e78197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 >> 172.21.15.89:0/739831142 conn(0x7f4e78075250 msgr2=0x7f4e780fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 shutdown_connections 2023-08-25T02:29:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:40.551+0000 7f4e7f047700 1 -- 172.21.15.89:0/739831142 wait complete. 2023-08-25T02:29:41.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:41 smithi121 ceph-mon[100099]: pgmap v98: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.2 MiB/s wr, 43 op/s 2023-08-25T02:29:41.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:41 smithi122 ceph-mon[100457]: pgmap v98: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.2 MiB/s wr, 43 op/s 2023-08-25T02:29:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:41 smithi089 ceph-mon[98732]: pgmap v98: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.2 MiB/s wr, 43 op/s 2023-08-25T02:29:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:43 smithi089 ceph-mon[98732]: pgmap v99: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 MiB/s wr, 54 op/s 2023-08-25T02:29:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:43 smithi121 ceph-mon[100099]: pgmap v99: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 MiB/s wr, 54 op/s 2023-08-25T02:29:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:43 smithi122 ceph-mon[100457]: pgmap v99: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 MiB/s wr, 54 op/s 2023-08-25T02:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:43300000 of 50000000 tuples (86%) done (elapsed 183.71 s, remaining 28.43 s) 2023-08-25T02:29:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:45 smithi089 ceph-mon[98732]: pgmap v100: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 15 op/s 2023-08-25T02:29:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:45 smithi121 ceph-mon[100099]: pgmap v100: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 15 op/s 2023-08-25T02:29:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:45 smithi122 ceph-mon[100457]: pgmap v100: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 15 op/s 2023-08-25T02:29:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:47 smithi089 ceph-mon[98732]: pgmap v101: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 23 MiB/s wr, 25 op/s 2023-08-25T02:29:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:47 smithi121 ceph-mon[100099]: pgmap v101: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 23 MiB/s wr, 25 op/s 2023-08-25T02:29:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:47 smithi122 ceph-mon[100457]: pgmap v101: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 23 MiB/s wr, 25 op/s 2023-08-25T02:29:48.807 INFO:teuthology.orchestra.run.smithi089.stderr:43400000 of 50000000 tuples (86%) done (elapsed 188.11 s, remaining 28.61 s) 2023-08-25T02:29:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:43500000 of 50000000 tuples (87%) done (elapsed 188.18 s, remaining 28.12 s) 2023-08-25T02:29:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:49 smithi089 ceph-mon[98732]: pgmap v102: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:49 smithi121 ceph-mon[100099]: pgmap v102: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:49.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:49 smithi122 ceph-mon[100457]: pgmap v102: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:51.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:51 smithi121 ceph-mon[100099]: pgmap v103: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:51 smithi122 ceph-mon[100457]: pgmap v103: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:51 smithi089 ceph-mon[98732]: pgmap v103: 97 pgs: 97 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 MiB/s wr, 20 op/s 2023-08-25T02:29:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.669+0000 7f5b3c56b700 1 -- 172.21.15.89:0/1550410183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b340ff9b0 msgr2=0x7f5b340ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.669+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/1550410183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b340ff9b0 0x7f5b340ffe30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009990 tx=0x7f5b30009ca0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- 172.21.15.89:0/1550410183 shutdown_connections 2023-08-25T02:29:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/1550410183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b341057b0 0x7f5b34105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/1550410183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b34100370 0x7f5b3410c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/1550410183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b340ff9b0 0x7f5b340ffe30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- 172.21.15.89:0/1550410183 >> 172.21.15.89:0/1550410183 conn(0x7f5b340776f0 msgr2=0x7f5b34077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- 172.21.15.89:0/1550410183 shutdown_connections 2023-08-25T02:29:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- 172.21.15.89:0/1550410183 wait complete. 2023-08-25T02:29:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 Processor -- start 2023-08-25T02:29:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- start start 2023-08-25T02:29:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b34100370 0x7f5b34197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 0x7f5b341a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b341a2e10 con 0x7f5b341057b0 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b341a2f80 con 0x7f5b34100370 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.670+0000 7f5b3c56b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b341a3140 con 0x7f5b340ff9b0 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40910/0 (socket says 172.21.15.89:40910) 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3ab08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 0x7f5b341a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 -- 172.21.15.89:0/604948119 learned_addr learned my addr 172.21.15.89:0/604948119 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b39b06700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b34100370 0x7f5b34197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 -- 172.21.15.89:0/604948119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b34100370 msgr2=0x7f5b34197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b34100370 0x7f5b34197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 -- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 msgr2=0x7f5b341a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 0x7f5b341a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 -- 172.21.15.89:0/604948119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300095f0 con 0x7f5b340ff9b0 2023-08-25T02:29:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 0x7f5b341a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3a307700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e9a0 tx=0x7f5b2400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b2b7fe700 1 -- 172.21.15.89:0/604948119 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24009790 con 0x7f5b340ff9b0 2023-08-25T02:29:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b341a3420 con 0x7f5b340ff9b0 2023-08-25T02:29:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b341a3980 con 0x7f5b340ff9b0 2023-08-25T02:29:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b2b7fe700 1 -- 172.21.15.89:0/604948119 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b24010430 con 0x7f5b340ff9b0 2023-08-25T02:29:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.671+0000 7f5b2b7fe700 1 -- 172.21.15.89:0/604948119 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24018680 con 0x7f5b340ff9b0 2023-08-25T02:29:53.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.674+0000 7f5b2b7fe700 1 -- 172.21.15.89:0/604948119 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f5b2400ca10 con 0x7f5b340ff9b0 2023-08-25T02:29:53.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.675+0000 7f5b2b7fe700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 0x7f5b200a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.675+0000 7f5b39b06700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 0x7f5b200a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.675+0000 7f5b2b7fe700 1 -- 172.21.15.89:0/604948119 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f5b24014070 con 0x7f5b340ff9b0 2023-08-25T02:29:53.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.675+0000 7f5b4870fb80 1 Processor -- start 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 -- start start 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b39b06700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 0x7f5b200a2580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b34069970 tx=0x7f5b30005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd31450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc99750 con 0x55cd9cd31990 2023-08-25T02:29:53.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc8a720 con 0x55cd9cd37860 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc88040 con 0x55cd9cd36f40 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3ab08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40918/0 (socket says 172.21.15.89:40918) 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3ab08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48986/0 (socket says 172.21.15.89:48986) 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 -- 172.21.15.89:0/216288778 learned_addr learned my addr 172.21.15.89:0/216288778 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b39b06700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd31450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 -- 172.21.15.89:0/216288778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 msgr2=0x55cd9cd31450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd31450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 -- 172.21.15.89:0/216288778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 msgr2=0x55cd9cd31dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 -- 172.21.15.89:0/216288778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd9cb035d0 con 0x55cd9cd36f40 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b39b06700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd31450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.676+0000 7f5b3a307700 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5b24010c20 tx=0x7f5b240987c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324dcfd25bd7813f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b28ff9700 1 -- 172.21.15.89:0/216288778 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2409cc10 con 0x55cd9cd36f40 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b28ff9700 1 -- 172.21.15.89:0/216288778 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b240a0bf0 con 0x55cd9cd36f40 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b28ff9700 1 -- 172.21.15.89:0/216288778 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240982d0 con 0x55cd9cd36f40 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 -- 172.21.15.89:0/216288778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 msgr2=0x55cd9cd37320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5b24010c20 tx=0x7f5b240987c0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 -- 172.21.15.89:0/216288778 shutdown_connections 2023-08-25T02:29:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd31990 0x55cd9cd31dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd37320 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/216288778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd31450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 -- 172.21.15.89:0/216288778 >> 172.21.15.89:0/216288778 conn(0x55cd9cca6080 msgr2=0x55cd9cd2bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 -- 172.21.15.89:0/216288778 shutdown_connections 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.677+0000 7f5b4870fb80 1 -- 172.21.15.89:0/216288778 wait complete. 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 -- start start 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 Processor -- start 2023-08-25T02:29:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd3a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd3af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc83930 con 0x55cd9cd36f40 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc21b50 con 0x55cd9cd37860 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cc21850 con 0x55cd9cd31990 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b4870fb80 10 client.0 _ll_register_callbacks cb 0x55cd9cb6ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b39b06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd3a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:40924/0 (socket says 172.21.15.89:40924) 2023-08-25T02:29:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 -- 172.21.15.89:0/594100012 learned_addr learned my addr 172.21.15.89:0/594100012 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b39b06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd3a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48990/0 (socket says 172.21.15.89:48990) 2023-08-25T02:29:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd3af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 -- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 msgr2=0x55cd9cd3af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd3af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 -- 172.21.15.89:0/594100012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 msgr2=0x55cd9cd3a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd3a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.678+0000 7f5b3a307700 1 -- 172.21.15.89:0/594100012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c003a00 con 0x55cd9cd31990 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b3a307700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009230 tx=0x7f5b24000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcda6e18dd7d6d22 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2409cc90 con 0x55cd9cd31990 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd9cd01160 con 0x55cd9cd31990 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f5b240982d0 con 0x55cd9cd31990 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b24098490 con 0x55cd9cd31990 2023-08-25T02:29:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 10 client.25213 fetch_fsmap learned FSMap version 30 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f5b341a3980 con 0x55cd9cd31990 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cd9cd01160 con 0x55cd9cd31990 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 30) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5b24097d40 con 0x55cd9cd31990 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 10 client.25213 fetch_fsmap finished waiting for FSMap version 30 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 10 client.25213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 20 client.25213 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cd4aee0 0x55cd9cd4d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 4 client.25213 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.679+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd9cc29b40 con 0x55cd9cd4aee0 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.680+0000 7f5b39b06700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cd4aee0 0x55cd9cd4d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.680+0000 7f5b1dffb700 10 client.25213 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.680+0000 7f5b39b06700 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cd4aee0 0x55cd9cd4d300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c4cc90499112022 server_cookie=f08c2c9543592c98 in_seq=0 out_seq=0 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b1dffb700 1 -- 172.21.15.89:0/594100012 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cd9cc29b40 con 0x55cd9cd4aee0 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b1dffb700 10 client.25213 handle_command_reply: tid=0 2023-08-25T02:29:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 client.25213 shutdown 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 msgr2=0x55cd9cd3d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009230 tx=0x7f5b24000bc0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 shutdown_connections 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cd4aee0 0x55cd9cd4d300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd37860 0x55cd9cd3af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd31990 0x55cd9cd3d6b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/594100012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd3a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b1dffb700 0 client.25213 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 >> 172.21.15.89:0/594100012 conn(0x55cd9cca6080 msgr2=0x55cd9cd26820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 shutdown_connections 2023-08-25T02:29:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 1 -- 172.21.15.89:0/594100012 wait complete. 2023-08-25T02:29:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.681+0000 7f5b4870fb80 20 client.25213 trim_cache size 0 max 16384 2023-08-25T02:29:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.695+0000 7f5b4870fb80 1 Processor -- start 2023-08-25T02:29:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 -- start start 2023-08-25T02:29:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd31d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd42240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cb058c0 con 0x55cd9cd37860 2023-08-25T02:29:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cd3b540 con 0x55cd9cd31990 2023-08-25T02:29:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cd2cfe0 con 0x55cd9cd36f40 2023-08-25T02:29:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49000/0 (socket says 172.21.15.89:49000) 2023-08-25T02:29:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 -- 172.21.15.89:0/869427224 learned_addr learned my addr 172.21.15.89:0/869427224 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b39b06700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd42240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3a307700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd31d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 -- 172.21.15.89:0/869427224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 msgr2=0x55cd9cd42240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd42240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 -- 172.21.15.89:0/869427224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 msgr2=0x55cd9cd31d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd31d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 -- 172.21.15.89:0/869427224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd9caafd80 con 0x55cd9cd37860 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3a307700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd31d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.696+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00eaf0 tx=0x7f5b2c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc7dc5541889323 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b297fa700 1 -- 172.21.15.89:0/869427224 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00e640 con 0x55cd9cd37860 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b297fa700 1 -- 172.21.15.89:0/869427224 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b2c00f040 con 0x55cd9cd37860 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b297fa700 1 -- 172.21.15.89:0/869427224 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c007a20 con 0x55cd9cd37860 2023-08-25T02:29:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- 172.21.15.89:0/869427224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 msgr2=0x55cd9cd469a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00eaf0 tx=0x7f5b2c009920 comp rx=0 tx=0).stop 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- 172.21.15.89:0/869427224 shutdown_connections 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd31d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd37860 0x55cd9cd469a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/869427224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd36f40 0x55cd9cd42240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- 172.21.15.89:0/869427224 >> 172.21.15.89:0/869427224 conn(0x55cd9cca6080 msgr2=0x55cd9cd2bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- 172.21.15.89:0/869427224 shutdown_connections 2023-08-25T02:29:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- 172.21.15.89:0/869427224 wait complete. 2023-08-25T02:29:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.697+0000 7f5b4870fb80 1 -- start start 2023-08-25T02:29:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 Processor -- start 2023-08-25T02:29:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd33f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 0x55cd9cd32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cd33300 con 0x55cd9cd36f40 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cd33470 con 0x55cd9cd31990 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd9cd33630 con 0x55cd9cd37860 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b4870fb80 10 client.0 _ll_register_callbacks cb 0x7f5b2c009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:29:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49004/0 (socket says 172.21.15.89:49004) 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 -- 172.21.15.89:0/3455429823 learned_addr learned my addr 172.21.15.89:0/3455429823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 0x55cd9cd32dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b3a307700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd33f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 -- 172.21.15.89:0/3455429823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 msgr2=0x55cd9cd32dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 0x55cd9cd32dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 -- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 msgr2=0x55cd9cd33f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd33f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 -- 172.21.15.89:0/3455429823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c00e7a0 con 0x55cd9cd36f40 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b3ab08700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 0x55cd9cd32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b39b06700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5b300098e0 tx=0x7f5b30009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6373ea3d1087d72 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.698+0000 7f5b1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b30034c10 con 0x55cd9cd36f40 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd9cd24600 con 0x55cd9cd36f40 2023-08-25T02:29:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f5b30034d70 con 0x55cd9cd36f40 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b30034070 con 0x55cd9cd36f40 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 10 client.15597 fetch_fsmap learned FSMap version 30 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cd9ca98c00 con 0x55cd9cd36f40 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cd9cd24600 con 0x55cd9cd36f40 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 30) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5b30004b10 con 0x55cd9cd36f40 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 10 client.15597 fetch_fsmap finished waiting for FSMap version 30 2023-08-25T02:29:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 20 client.15597 populate_metadata read hostname 'smithi089' 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cda4290 0x55cd9cda66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 4 client.15597 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cd9cc7c910 con 0x55cd9cda4290 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.699+0000 7f5b3a307700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cda4290 0x55cd9cda66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:29:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.700+0000 7f5b1dffb700 10 client.15597 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:53.700+0000 7f5b3a307700 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cda4290 0x55cd9cda66b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98794902236dca2b server_cookie=4d594212089da524 in_seq=0 out_seq=0 2023-08-25T02:29:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:53 smithi089 ceph-mon[98732]: pgmap v104: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 28 op/s 2023-08-25T02:29:53.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:53 smithi121 ceph-mon[100099]: pgmap v104: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 28 op/s 2023-08-25T02:29:53.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:53 smithi122 ceph-mon[100457]: pgmap v104: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 27 MiB/s wr, 28 op/s 2023-08-25T02:29:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b1dffb700 1 -- 172.21.15.89:0/3455429823 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cd9cc7c910 con 0x55cd9cda4290 2023-08-25T02:29:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b1dffb700 10 client.15597 handle_command_reply: tid=0 2023-08-25T02:29:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 client.15597 shutdown 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 msgr2=0x55cd9cd32880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5b300098e0 tx=0x7f5b30009ca0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 shutdown_connections 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55cd9cda4290 0x55cd9cda66b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cd9cd31990 0x55cd9cd33f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b1dffb700 0 client.15597 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cd9cd37860 0x55cd9cd32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 --2- 172.21.15.89:0/3455429823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cd9cd36f40 0x55cd9cd32880 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 >> 172.21.15.89:0/3455429823 conn(0x55cd9cca6080 msgr2=0x55cd9cd47420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 shutdown_connections 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 1 -- 172.21.15.89:0/3455429823 wait complete. 2023-08-25T02:29:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.071+0000 7f5b4870fb80 20 client.15597 trim_cache size 0 max 16384 2023-08-25T02:29:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:29:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:29:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:29:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:29:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 msgr2=0x7f5b200a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 0x7f5b200a2580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b34069970 tx=0x7f5b30005fd0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 msgr2=0x7f5b341979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e9a0 tx=0x7f5b2400ecb0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 shutdown_connections 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f5b200a00c0 0x7f5b200a2580 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b341057b0 0x7f5b341a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b340ff9b0 0x7f5b341979a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 --2- 172.21.15.89:0/604948119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b34100370 0x7f5b34197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:29:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 >> 172.21.15.89:0/604948119 conn(0x7f5b340776f0 msgr2=0x7f5b340fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:29:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 shutdown_connections 2023-08-25T02:29:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:29:54.072+0000 7f5b3c56b700 1 -- 172.21.15.89:0/604948119 wait complete. 2023-08-25T02:29:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:55 smithi089 ceph-mon[98732]: pgmap v105: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 18 op/s 2023-08-25T02:29:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:55 smithi121 ceph-mon[100099]: pgmap v105: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 18 op/s 2023-08-25T02:29:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:55 smithi122 ceph-mon[100457]: pgmap v105: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 18 op/s 2023-08-25T02:29:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:57 smithi089 ceph-mon[98732]: pgmap v106: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 24 op/s 2023-08-25T02:29:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:57 smithi121 ceph-mon[100099]: pgmap v106: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 24 op/s 2023-08-25T02:29:57.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:57 smithi122 ceph-mon[100457]: pgmap v106: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 24 op/s 2023-08-25T02:29:58.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:58 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:58.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:58 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:58.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:58 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:29:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:29:59 smithi089 ceph-mon[98732]: pgmap v107: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 14 op/s 2023-08-25T02:29:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:29:59 smithi121 ceph-mon[100099]: pgmap v107: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 14 op/s 2023-08-25T02:29:59.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:29:59 smithi122 ceph-mon[100457]: pgmap v107: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.7 MiB/s wr, 14 op/s 2023-08-25T02:30:00.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:00 smithi089 ceph-mon[98732]: overall HEALTH_OK 2023-08-25T02:30:00.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:00 smithi121 ceph-mon[100099]: overall HEALTH_OK 2023-08-25T02:30:00.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:00 smithi122 ceph-mon[100457]: overall HEALTH_OK 2023-08-25T02:30:01.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:01 smithi089 ceph-mon[98732]: pgmap v108: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.0 MiB/s wr, 14 op/s 2023-08-25T02:30:01.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:01 smithi121 ceph-mon[100099]: pgmap v108: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.0 MiB/s wr, 14 op/s 2023-08-25T02:30:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:01 smithi122 ceph-mon[100457]: pgmap v108: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.0 MiB/s wr, 14 op/s 2023-08-25T02:30:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:03 smithi089 ceph-mon[98732]: pgmap v109: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.1 MiB/s wr, 30 op/s 2023-08-25T02:30:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:03 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:03 smithi121 ceph-mon[100099]: pgmap v109: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.1 MiB/s wr, 30 op/s 2023-08-25T02:30:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:03 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:03.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:03 smithi122 ceph-mon[100457]: pgmap v109: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.1 MiB/s wr, 30 op/s 2023-08-25T02:30:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:03 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:05 smithi089 ceph-mon[98732]: pgmap v110: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 21 op/s 2023-08-25T02:30:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:05 smithi121 ceph-mon[100099]: pgmap v110: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 21 op/s 2023-08-25T02:30:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:05 smithi122 ceph-mon[100457]: pgmap v110: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 21 op/s 2023-08-25T02:30:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 -- 172.21.15.89:0/3481398321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 msgr2=0x7fa3d80fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 --2- 172.21.15.89:0/3481398321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d80fff00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40099c0 tx=0x7fa3c4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 -- 172.21.15.89:0/3481398321 shutdown_connections 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 --2- 172.21.15.89:0/3481398321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 0x7fa3d810c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 --2- 172.21.15.89:0/3481398321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 0x7fa3d8105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 --2- 172.21.15.89:0/3481398321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d80fff00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 -- 172.21.15.89:0/3481398321 >> 172.21.15.89:0/3481398321 conn(0x7fa3d8075250 msgr2=0x7fa3d8075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 -- 172.21.15.89:0/3481398321 shutdown_connections 2023-08-25T02:30:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.190+0000 7fa3def36700 1 -- 172.21.15.89:0/3481398321 wait complete. 2023-08-25T02:30:07.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 Processor -- start 2023-08-25T02:30:07.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 -- start start 2023-08-25T02:30:07.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 0x7fa3d8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 0x7fa3d81a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81a2de0 con 0x7fa3d80ffa80 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81a2f50 con 0x7fa3d8100440 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3def36700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81a3110 con 0x7fa3d8105840 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49016/0 (socket says 172.21.15.89:49016) 2023-08-25T02:30:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3686787416 learned_addr learned my addr 172.21.15.89:0/3686787416 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 0x7fa3d81a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 0x7fa3d8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3686787416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 msgr2=0x7fa3d81a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 0x7fa3d81a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3686787416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 msgr2=0x7fa3d8197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 0x7fa3d8197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3686787416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c40095f0 con 0x7fa3d80ffa80 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.191+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e9a0 tx=0x7fa3cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.192+0000 7fa3d5ffb700 1 -- 172.21.15.89:0/3686787416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc009790 con 0x7fa3d80ffa80 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.192+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81a33f0 con 0x7fa3d80ffa80 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.192+0000 7fa3d5ffb700 1 -- 172.21.15.89:0/3686787416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3cc010430 con 0x7fa3d80ffa80 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.192+0000 7fa3d5ffb700 1 -- 172.21.15.89:0/3686787416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc018680 con 0x7fa3d80ffa80 2023-08-25T02:30:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.192+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81a38d0 con 0x7fa3d80ffa80 2023-08-25T02:30:07.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.194+0000 7fa3d5ffb700 1 -- 172.21.15.89:0/3686787416 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fa3cc0105a0 con 0x7fa3d80ffa80 2023-08-25T02:30:07.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.194+0000 7fa3d5ffb700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 0x7fa3c80a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.195+0000 7fa3d5ffb700 1 -- 172.21.15.89:0/3686787416 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa3cc014070 con 0x7fa3d80ffa80 2023-08-25T02:30:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.195+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 0x7fa3c80a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.195+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 0x7fa3c80a2600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009990 tx=0x7fa3c4005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 Processor -- start 2023-08-25T02:30:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 -- start start 2023-08-25T02:30:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c1fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 0x5557f4c23eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b83750 con 0x5557f4c243f0 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b74720 con 0x5557f4c1fa50 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b72040 con 0x5557f4c20370 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 0x5557f4c23eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49028/0 (socket says 172.21.15.89:49028) 2023-08-25T02:30:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/3891723601 learned_addr learned my addr 172.21.15.89:0/3891723601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c1fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/3891723601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 msgr2=0x5557f4c23eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 0x5557f4c23eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/3891723601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 msgr2=0x5557f4c1fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c1fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/3891723601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557f49ed5d0 con 0x5557f4c243f0 2023-08-25T02:30:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 0x5557f4c23eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.196+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011680 tx=0x7fa3d0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d12060d2065daa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3c2ffd700 1 -- 172.21.15.89:0/3891723601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000e040 con 0x5557f4c243f0 2023-08-25T02:30:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3c2ffd700 1 -- 172.21.15.89:0/3891723601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3d000f040 con 0x5557f4c243f0 2023-08-25T02:30:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3c2ffd700 1 -- 172.21.15.89:0/3891723601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0004860 con 0x5557f4c243f0 2023-08-25T02:30:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3891723601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 msgr2=0x5557f4c24850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011680 tx=0x7fa3d0009920 comp rx=0 tx=0).stop 2023-08-25T02:30:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3891723601 shutdown_connections 2023-08-25T02:30:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c1fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c243f0 0x5557f4c24850 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3891723601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c20370 0x5557f4c23eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3891723601 >> 172.21.15.89:0/3891723601 conn(0x5557f4b90080 msgr2=0x5557f4c15d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3891723601 shutdown_connections 2023-08-25T02:30:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.197+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3891723601 wait complete. 2023-08-25T02:30:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 -- start start 2023-08-25T02:30:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 Processor -- start 2023-08-25T02:30:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 0x5557f4c2bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c20370 0x5557f4c28e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b6d930 con 0x5557f4c1fa50 2023-08-25T02:30:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b0bb50 con 0x5557f4c20370 2023-08-25T02:30:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4b0b850 con 0x5557f4c243f0 2023-08-25T02:30:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3eb0dab80 10 client.0 _ll_register_callbacks cb 0x5557f4a55a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3dccd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 0x5557f4c2bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.198+0000 7fa3dd4d3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dccd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 0x5557f4c2bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49030/0 (socket says 172.21.15.89:49030) 2023-08-25T02:30:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c20370 0x5557f4c28e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dccd2700 1 -- 172.21.15.89:0/1752495928 learned_addr learned my addr 172.21.15.89:0/1752495928 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c20370 msgr2=0x5557f4c28e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c20370 0x5557f4c28e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/1752495928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 msgr2=0x5557f4c2bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 0x5557f4c2bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 -- 172.21.15.89:0/1752495928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d810cee0 con 0x5557f4c243f0 2023-08-25T02:30:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c294d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011650 tx=0x7fa3d0006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b472adce5819af7a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:30:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000e070 con 0x5557f4c243f0 2023-08-25T02:30:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa3d0003a00 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557f4beb160 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3d0003bc0 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3eb0dab80 10 client.25231 fetch_fsmap learned FSMap version 32 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.199+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7fa3d0011330 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5557f4beb160 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 32) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa3d0024030 con 0x5557f4c243f0 2023-08-25T02:30:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 10 client.25231 fetch_fsmap finished waiting for FSMap version 32 2023-08-25T02:30:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 10 client.25231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 20 client.25231 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c39630 0x5557f4c3ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 4 client.25231 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557f4b13b40 con 0x5557f4c39630 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c39630 0x5557f4c3ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3b7fff700 10 client.25231 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.200+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c39630 0x5557f4c3ba50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a418b3c41323f82b server_cookie=7fcc628f0785b751 in_seq=0 out_seq=0 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3b7fff700 1 -- 172.21.15.89:0/1752495928 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5557f4b13b40 con 0x5557f4c39630 2023-08-25T02:30:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3b7fff700 10 client.25231 handle_command_reply: tid=0 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 client.25231 shutdown 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 msgr2=0x5557f4c294d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c294d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011650 tx=0x7fa3d0006000 comp rx=0 tx=0).stop 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 shutdown_connections 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c39630 0x5557f4c3ba50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c294d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3b7fff700 0 client.25231 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c1fa50 0x5557f4c2bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/1752495928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c20370 0x5557f4c28e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 >> 172.21.15.89:0/1752495928 conn(0x5557f4b90080 msgr2=0x5557f4c247d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 shutdown_connections 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/1752495928 wait complete. 2023-08-25T02:30:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.201+0000 7fa3eb0dab80 20 client.25231 trim_cache size 0 max 16384 2023-08-25T02:30:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 Processor -- start 2023-08-25T02:30:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 -- start start 2023-08-25T02:30:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c30720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c22400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f49ef8c0 con 0x5557f4c20370 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4c298a0 con 0x5557f4c1fa50 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4c162a0 con 0x5557f4c243f0 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49040/0 (socket says 172.21.15.89:49040) 2023-08-25T02:30:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 -- 172.21.15.89:0/118343074 learned_addr learned my addr 172.21.15.89:0/118343074 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c22400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c30720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 -- 172.21.15.89:0/118343074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 msgr2=0x5557f4c22400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c22400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 -- 172.21.15.89:0/118343074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 msgr2=0x5557f4c30720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c30720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 -- 172.21.15.89:0/118343074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557f4c0fb40 con 0x5557f4c20370 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.216+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40099c0 tx=0x7fa3c4022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd0dcde4b67cd21 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3c37fe700 1 -- 172.21.15.89:0/118343074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c402c030 con 0x5557f4c20370 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3c37fe700 1 -- 172.21.15.89:0/118343074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3c402d040 con 0x5557f4c20370 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3c37fe700 1 -- 172.21.15.89:0/118343074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400f6a0 con 0x5557f4c20370 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/118343074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 msgr2=0x5557f4c257e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40099c0 tx=0x7fa3c4022000 comp rx=0 tx=0).stop 2023-08-25T02:30:07.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/118343074 shutdown_connections 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c1fa50 0x5557f4c30720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c243f0 0x5557f4c22400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/118343074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c257e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/118343074 >> 172.21.15.89:0/118343074 conn(0x5557f4b90080 msgr2=0x5557f4c34e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/118343074 shutdown_connections 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.217+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/118343074 wait complete. 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 -- start start 2023-08-25T02:30:07.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 Processor -- start 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 0x5557f4c28950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 0x5557f4c27b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4c29190 con 0x5557f4c20370 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4c28080 con 0x5557f4c243f0 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f4c28240 con 0x5557f4c1fa50 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3eb0dab80 10 client.0 _ll_register_callbacks cb 0x7fa3cc005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:07.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3dccd2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 0x5557f4c28950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3d7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49046/0 (socket says 172.21.15.89:49046) 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.218+0000 7fa3d7fff700 1 -- 172.21.15.89:0/3825299844 learned_addr learned my addr 172.21.15.89:0/3825299844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 0x5557f4c27b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 msgr2=0x5557f4c27b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 0x5557f4c27b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3825299844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 msgr2=0x5557f4c27600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 -- 172.21.15.89:0/3825299844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c4003bb0 con 0x5557f4c1fa50 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dd4d3700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 0x5557f4c27b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3dccd2700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 0x5557f4c28950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc095420 tx=0x7fa3cc010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b702f5316a55fdb9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:30:07.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc098880 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fa3cc0a0d30 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557f4c0e600 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3cc099180 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3eb0dab80 10 client.25237 fetch_fsmap learned FSMap version 32 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5557f4999d80 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3cc064030 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 32) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa3cc0a08a0 con 0x5557f4c1fa50 2023-08-25T02:30:07.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3eb0dab80 10 client.25237 fetch_fsmap finished waiting for FSMap version 32 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.219+0000 7fa3eb0dab80 10 client.25237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3eb0dab80 20 client.25237 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c8e770 0x5557f4c90b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3eb0dab80 4 client.25237 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5557f4b66910 con 0x5557f4c8e770 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c8e770 0x5557f4c90b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3d7fff700 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c8e770 0x5557f4c90b90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2ac441b0ca8e97 server_cookie=2a48a3f664e7376d in_seq=0 out_seq=0 2023-08-25T02:30:07.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.220+0000 7fa3b7fff700 10 client.25237 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3b7fff700 1 -- 172.21.15.89:0/3825299844 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5557f4b66910 con 0x5557f4c8e770 2023-08-25T02:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3b7fff700 10 client.25237 handle_command_reply: tid=0 2023-08-25T02:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 client.25237 shutdown 2023-08-25T02:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 msgr2=0x5557f4c28950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 0x5557f4c28950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc095420 tx=0x7fa3cc010c80 comp rx=0 tx=0).stop 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 shutdown_connections 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5557f4c8e770 0x5557f4c90b90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557f4c243f0 0x5557f4c27b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3b7fff700 0 client.25237 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5557f4c1fa50 0x5557f4c28950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 --2- 172.21.15.89:0/3825299844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5557f4c20370 0x5557f4c27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 >> 172.21.15.89:0/3825299844 conn(0x5557f4b90080 msgr2=0x5557f4c22400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 shutdown_connections 2023-08-25T02:30:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.798+0000 7fa3eb0dab80 1 -- 172.21.15.89:0/3825299844 wait complete. 2023-08-25T02:30:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3eb0dab80 20 client.25237 trim_cache size 0 max 16384 2023-08-25T02:30:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:30:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:30:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:30:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:30:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 msgr2=0x7fa3c80a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 0x7fa3c80a2600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009990 tx=0x7fa3c4005fb0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 msgr2=0x7fa3d8197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e9a0 tx=0x7fa3cc00ecb0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 shutdown_connections 2023-08-25T02:30:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa3c80a0140 0x7fa3c80a2600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3d8105840 0x7fa3d81a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d80ffa80 0x7fa3d8197970 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 --2- 172.21.15.89:0/3686787416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3d8100440 0x7fa3d8197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 >> 172.21.15.89:0/3686787416 conn(0x7fa3d8075250 msgr2=0x7fa3d80fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 shutdown_connections 2023-08-25T02:30:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:07.799+0000 7fa3def36700 1 -- 172.21.15.89:0/3686787416 wait complete. 2023-08-25T02:30:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:07 smithi089 ceph-mon[98732]: pgmap v111: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 KiB/s wr, 26 op/s 2023-08-25T02:30:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:07 smithi121 ceph-mon[100099]: pgmap v111: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 KiB/s wr, 26 op/s 2023-08-25T02:30:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:07 smithi122 ceph-mon[100457]: pgmap v111: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 KiB/s wr, 26 op/s 2023-08-25T02:30:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:09 smithi089 ceph-mon[98732]: pgmap v112: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:09 smithi121 ceph-mon[100099]: pgmap v112: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:09 smithi122 ceph-mon[100457]: pgmap v112: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:10.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:10 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:10 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:10 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:11 smithi089 ceph-mon[98732]: pgmap v113: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:11 smithi121 ceph-mon[100099]: pgmap v113: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:11 smithi122 ceph-mon[100457]: pgmap v113: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 31 KiB/s wr, 20 op/s 2023-08-25T02:30:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:13 smithi089 ceph-mon[98732]: pgmap v114: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 25 op/s 2023-08-25T02:30:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:13 smithi121 ceph-mon[100099]: pgmap v114: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 25 op/s 2023-08-25T02:30:13.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:13 smithi122 ceph-mon[100457]: pgmap v114: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 25 op/s 2023-08-25T02:30:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:15 smithi089 ceph-mon[98732]: pgmap v115: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-08-25T02:30:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:15 smithi121 ceph-mon[100099]: pgmap v115: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-08-25T02:30:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:15 smithi122 ceph-mon[100457]: pgmap v115: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-08-25T02:30:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:17 smithi089 ceph-mon[98732]: pgmap v116: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-08-25T02:30:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:17 smithi121 ceph-mon[100099]: pgmap v116: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-08-25T02:30:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:17 smithi122 ceph-mon[100457]: pgmap v116: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 14 op/s 2023-08-25T02:30:19.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:19 smithi089 ceph-mon[98732]: pgmap v117: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:19 smithi121 ceph-mon[100099]: pgmap v117: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:19 smithi122 ceph-mon[100457]: pgmap v117: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 -- 172.21.15.89:0/3027424337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17001057e0 msgr2=0x7f1700105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 --2- 172.21.15.89:0/3027424337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17001057e0 0x7f1700105bc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009990 tx=0x7f16f0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 -- 172.21.15.89:0/3027424337 shutdown_connections 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 --2- 172.21.15.89:0/3027424337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1700100190 0x7f170010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 --2- 172.21.15.89:0/3027424337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17001057e0 0x7f1700105bc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 --2- 172.21.15.89:0/3027424337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 0x7f17000ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.917+0000 7f1707529700 1 -- 172.21.15.89:0/3027424337 >> 172.21.15.89:0/3027424337 conn(0x7f17000fb260 msgr2=0x7f17000fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- 172.21.15.89:0/3027424337 shutdown_connections 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- 172.21.15.89:0/3027424337 wait complete. 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 Processor -- start 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- start start 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 0x7f1700197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001057e0 0x7f17001a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a2df0 con 0x7f1700100190 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a2f60 con 0x7f17001057e0 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1707529700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a3120 con 0x7f17000ff7d0 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f17052c5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 0x7f1700197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.918+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49060/0 (socket says 172.21.15.89:49060) 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 -- 172.21.15.89:0/2675528457 learned_addr learned my addr 172.21.15.89:0/2675528457 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1705ac6700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001057e0 0x7f17001a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 -- 172.21.15.89:0/2675528457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 msgr2=0x7f1700197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 0x7f1700197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 -- 172.21.15.89:0/2675528457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001057e0 msgr2=0x7f17001a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001057e0 0x7f17001a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 -- 172.21.15.89:0/2675528457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00095f0 con 0x7f1700100190 2023-08-25T02:30:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1704ac4700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e9a0 tx=0x7f16fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f16f67fc700 1 -- 172.21.15.89:0/2675528457 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc009790 con 0x7f1700100190 2023-08-25T02:30:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f16f67fc700 1 -- 172.21.15.89:0/2675528457 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16fc010430 con 0x7f1700100190 2023-08-25T02:30:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17001a3400 con 0x7f1700100190 2023-08-25T02:30:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f16f67fc700 1 -- 172.21.15.89:0/2675528457 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc018680 con 0x7f1700100190 2023-08-25T02:30:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.919+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17001a3910 con 0x7f1700100190 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.921+0000 7f16f67fc700 1 -- 172.21.15.89:0/2675528457 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f16fc0105a0 con 0x7f1700100190 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.922+0000 7f16f67fc700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 0x7f16ec0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.922+0000 7f17052c5700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 0x7f16ec0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.922+0000 7f16f67fc700 1 -- 172.21.15.89:0/2675528457 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f16fc014070 con 0x7f1700100190 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.922+0000 7f17052c5700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 0x7f16ec0a2580 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16f000b2e0 tx=0x7f16f0005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 Processor -- start 2023-08-25T02:30:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 -- start start 2023-08-25T02:30:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 0x55f66d613450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 0x55f66d613dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d57b750 con 0x55f66d618f40 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d56c720 con 0x55f66d619860 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17136cdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d56a040 con 0x55f66d613990 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17052c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17052c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49066/0 (socket says 172.21.15.89:49066) 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f1705ac6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 0x55f66d613dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f17052c5700 1 -- 172.21.15.89:0/1660417470 learned_addr learned my addr 172.21.15.89:0/1660417470 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.923+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 0x55f66d613450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 -- 172.21.15.89:0/1660417470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 msgr2=0x55f66d613dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 0x55f66d613dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 -- 172.21.15.89:0/1660417470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 msgr2=0x55f66d613450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 0x55f66d613450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 -- 172.21.15.89:0/1660417470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f66d3e55d0 con 0x55f66d618f40 2023-08-25T02:30:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f1705ac6700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 0x55f66d613dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 0x55f66d613450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f17052c5700 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f16f0015000 tx=0x7f16f0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3702bd024f3aee server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f16eb7fe700 1 -- 172.21.15.89:0/1660417470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f002b070 con 0x55f66d618f40 2023-08-25T02:30:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f16eb7fe700 1 -- 172.21.15.89:0/1660417470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16f001e040 con 0x55f66d618f40 2023-08-25T02:30:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.924+0000 7f16eb7fe700 1 -- 172.21.15.89:0/1660417470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000f6c0 con 0x55f66d618f40 2023-08-25T02:30:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- 172.21.15.89:0/1660417470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 msgr2=0x55f66d619320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f16f0015000 tx=0x7f16f0020000 comp rx=0 tx=0).stop 2023-08-25T02:30:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- 172.21.15.89:0/1660417470 shutdown_connections 2023-08-25T02:30:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d613990 0x55f66d613dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d619320 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1660417470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d619860 0x55f66d613450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- 172.21.15.89:0/1660417470 >> 172.21.15.89:0/1660417470 conn(0x55f66d588080 msgr2=0x55f66d60dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- 172.21.15.89:0/1660417470 shutdown_connections 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- 172.21.15.89:0/1660417470 wait complete. 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 -- start start 2023-08-25T02:30:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.925+0000 7f17136cdb80 1 Processor -- start 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d61f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d61d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d565930 con 0x55f66d618f40 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d503b50 con 0x55f66d613990 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d503850 con 0x55f66d619860 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 10 client.0 _ll_register_callbacks cb 0x55f66d44da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49074/0 (socket says 172.21.15.89:49074) 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 -- 172.21.15.89:0/1855382902 learned_addr learned my addr 172.21.15.89:0/1855382902 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1705ac6700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d61d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17052c5700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d61f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 -- 172.21.15.89:0/1855382902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 msgr2=0x55f66d61d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d61d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 -- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 msgr2=0x55f66d61f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d61f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 -- 172.21.15.89:0/1855382902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17001a3910 con 0x55f66d618f40 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17052c5700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d61f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f16fc098c20 tx=0x7f16fc0996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf06e34e5e766d8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f16e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc098610 con 0x55f66d618f40 2023-08-25T02:30:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f16fc09cd50 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f66d5e3160 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16fc0a0d30 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 10 client.15627 fetch_fsmap learned FSMap version 33 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.926+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f170010cfa0 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f66d5e3160 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 33) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f16fc094d10 con 0x55f66d618f40 2023-08-25T02:30:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 10 client.15627 fetch_fsmap finished waiting for FSMap version 33 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 20 client.15627 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d62d160 0x55f66d62f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 4 client.15627 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f66d50bb40 con 0x55f66d62d160 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.927+0000 7f17052c5700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d62d160 0x55f66d62f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f16e8ff9700 10 client.15627 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17052c5700 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d62d160 0x55f66d62f580 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ca5e7dced9bde64 server_cookie=4b9ceeb6a985b869 in_seq=0 out_seq=0 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f16e8ff9700 1 -- 172.21.15.89:0/1855382902 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f66d50bb40 con 0x55f66d62d160 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f16e8ff9700 10 client.15627 handle_command_reply: tid=0 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 client.15627 shutdown 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 msgr2=0x55f66d61c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f16fc098c20 tx=0x7f16fc0996a0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 shutdown_connections 2023-08-25T02:30:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d62d160 0x55f66d62f580 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d61f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d61d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1855382902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d61c910 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f16e8ff9700 0 client.15627 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.928+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 >> 172.21.15.89:0/1855382902 conn(0x55f66d588080 msgr2=0x55f66d608820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.929+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 shutdown_connections 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.929+0000 7f17136cdb80 1 -- 172.21.15.89:0/1855382902 wait complete. 2023-08-25T02:30:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.929+0000 7f17136cdb80 20 client.15627 trim_cache size 0 max 16384 2023-08-25T02:30:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 Processor -- start 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 -- start start 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d62f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d618f40 0x55f66d6241e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d3e78c0 con 0x55f66d619860 2023-08-25T02:30:20.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d5e3160 con 0x55f66d613990 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17136cdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d60a560 con 0x55f66d618f40 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49084/0 (socket says 172.21.15.89:49084) 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 -- 172.21.15.89:0/1894920783 learned_addr learned my addr 172.21.15.89:0/1894920783 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f17052c5700 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d62f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1704ac4700 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d618f40 0x55f66d6241e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 -- 172.21.15.89:0/1894920783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d618f40 msgr2=0x55f66d6241e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d618f40 0x55f66d6241e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 -- 172.21.15.89:0/1894920783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 msgr2=0x55f66d62f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d62f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 -- 172.21.15.89:0/1894920783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f66d391d80 con 0x55f66d619860 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.944+0000 7f1705ac6700 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009f40 tx=0x7f16f8009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b9fff42bc14339 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f16ebfff700 1 -- 172.21.15.89:0/1894920783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8004db0 con 0x55f66d619860 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f16ebfff700 1 -- 172.21.15.89:0/1894920783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16f800f040 con 0x55f66d619860 2023-08-25T02:30:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f16ebfff700 1 -- 172.21.15.89:0/1894920783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8005020 con 0x55f66d619860 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 -- 172.21.15.89:0/1894920783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 msgr2=0x55f66d61f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009f40 tx=0x7f16f8009f70 comp rx=0 tx=0).stop 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 -- 172.21.15.89:0/1894920783 shutdown_connections 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d62f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d619860 0x55f66d61f230 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 --2- 172.21.15.89:0/1894920783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d618f40 0x55f66d6241e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 -- 172.21.15.89:0/1894920783 >> 172.21.15.89:0/1894920783 conn(0x55f66d588080 msgr2=0x55f66d60dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 -- 172.21.15.89:0/1894920783 shutdown_connections 2023-08-25T02:30:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.945+0000 7f17136cdb80 1 -- 172.21.15.89:0/1894920783 wait complete. 2023-08-25T02:30:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 -- start start 2023-08-25T02:30:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 Processor -- start 2023-08-25T02:30:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d615b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d614880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d614dc0 con 0x55f66d618f40 2023-08-25T02:30:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d614f30 con 0x55f66d613990 2023-08-25T02:30:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f66d6150f0 con 0x55f66d619860 2023-08-25T02:30:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17136cdb80 10 client.0 _ll_register_callbacks cb 0x7f16fc098880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1704ac4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49088/0 (socket says 172.21.15.89:49088) 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d614880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1704ac4700 1 -- 172.21.15.89:0/2028177901 learned_addr learned my addr 172.21.15.89:0/2028177901 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f17052c5700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d615b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 -- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 msgr2=0x55f66d615b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d615b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 -- 172.21.15.89:0/2028177901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 msgr2=0x55f66d6160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.946+0000 7f1705ac6700 1 -- 172.21.15.89:0/2028177901 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8009bf0 con 0x55f66d619860 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f1704ac4700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f1705ac6700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d614880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009790 tx=0x7f16f8004d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fb418d412c5b62a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f801f620 con 0x55f66d619860 2023-08-25T02:30:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f16f801fe10 con 0x55f66d619860 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f66d606600 con 0x55f66d619860 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f801f780 con 0x55f66d619860 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 10 client.25255 fetch_fsmap learned FSMap version 33 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f66d37ac00 con 0x55f66d619860 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f66d606600 con 0x55f66d619860 2023-08-25T02:30:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 33) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f16f802e070 con 0x55f66d619860 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 10 client.25255 fetch_fsmap finished waiting for FSMap version 33 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 10 client.25255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 20 client.25255 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.947+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d681dd0 0x55f66d6841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.948+0000 7f17136cdb80 4 client.25255 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.948+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f66d55e910 con 0x55f66d681dd0 2023-08-25T02:30:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.948+0000 7f17052c5700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d681dd0 0x55f66d6841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.948+0000 7f16e8ff9700 10 client.25255 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.948+0000 7f17052c5700 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d681dd0 0x55f66d6841f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad8bbb93ab637257 server_cookie=977003e31860af7f in_seq=0 out_seq=0 2023-08-25T02:30:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f16e8ff9700 1 -- 172.21.15.89:0/2028177901 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f66d55e910 con 0x55f66d681dd0 2023-08-25T02:30:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f16e8ff9700 10 client.25255 handle_command_reply: tid=0 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 client.25255 shutdown 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 msgr2=0x55f66d614880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d614880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009790 tx=0x7f16f8004d80 comp rx=0 tx=0).stop 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 shutdown_connections 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55f66d681dd0 0x55f66d6841f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f66d613990 0x55f66d615b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f66d619860 0x55f66d614880 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f16e8ff9700 0 client.25255 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 --2- 172.21.15.89:0/2028177901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f66d618f40 0x55f66d6160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 >> 172.21.15.89:0/2028177901 conn(0x55f66d588080 msgr2=0x55f66d616af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 shutdown_connections 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 1 -- 172.21.15.89:0/2028177901 wait complete. 2023-08-25T02:30:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.956+0000 7f17136cdb80 20 client.25255 trim_cache size 0 max 16384 2023-08-25T02:30:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:30:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:30:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:30:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:30:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 msgr2=0x7f16ec0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 0x7f16ec0a2580 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16f000b2e0 tx=0x7f16f0005fd0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 msgr2=0x7f1700197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e9a0 tx=0x7f16fc00ecb0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 shutdown_connections 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f16ec0a00c0 0x7f16ec0a2580 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001057e0 0x7f17001a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17000ff7d0 0x7f1700197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 --2- 172.21.15.89:0/2675528457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1700100190 0x7f1700197ec0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 >> 172.21.15.89:0/2675528457 conn(0x7f17000fb260 msgr2=0x7f17000fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 shutdown_connections 2023-08-25T02:30:20.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:20.957+0000 7f1707529700 1 -- 172.21.15.89:0/2675528457 wait complete. 2023-08-25T02:30:21.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:21 smithi121 ceph-mon[100099]: pgmap v118: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:21.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:21 smithi122 ceph-mon[100457]: pgmap v118: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:21 smithi089 ceph-mon[98732]: pgmap v118: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:23 smithi089 ceph-mon[98732]: pgmap v119: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-08-25T02:30:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:23 smithi121 ceph-mon[100099]: pgmap v119: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-08-25T02:30:23.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:23 smithi122 ceph-mon[100457]: pgmap v119: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-08-25T02:30:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:25 smithi089 ceph-mon[98732]: pgmap v120: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:25 smithi121 ceph-mon[100099]: pgmap v120: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:25 smithi122 ceph-mon[100457]: pgmap v120: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-08-25T02:30:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:27 smithi089 ceph-mon[98732]: pgmap v121: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 12 op/s 2023-08-25T02:30:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:27 smithi121 ceph-mon[100099]: pgmap v121: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 12 op/s 2023-08-25T02:30:27.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:27 smithi122 ceph-mon[100457]: pgmap v121: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 12 op/s 2023-08-25T02:30:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:29 smithi089 ceph-mon[98732]: pgmap v122: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:29 smithi121 ceph-mon[100099]: pgmap v122: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:29.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:29 smithi122 ceph-mon[100457]: pgmap v122: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:31 smithi121 ceph-mon[100099]: pgmap v123: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:31.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:31 smithi122 ceph-mon[100457]: pgmap v123: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:31 smithi089 ceph-mon[98732]: pgmap v123: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:30:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:33 smithi089 ceph-mon[98732]: pgmap v124: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-08-25T02:30:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:33 smithi121 ceph-mon[100099]: pgmap v124: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-08-25T02:30:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:33 smithi122 ceph-mon[100457]: pgmap v124: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-08-25T02:30:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.075+0000 7fef3aa6f700 1 -- 172.21.15.89:0/2744100144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 msgr2=0x7fef34105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.075+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/2744100144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef34105c20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fef280099c0 tx=0x7fef28009cd0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.075+0000 7fef3aa6f700 1 -- 172.21.15.89:0/2744100144 shutdown_connections 2023-08-25T02:30:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.075+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/2744100144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef34100440 0x7fef3410c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.075+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/2744100144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef34105c20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/2744100144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef340ffa80 0x7fef340fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- 172.21.15.89:0/2744100144 >> 172.21.15.89:0/2744100144 conn(0x7fef34075250 msgr2=0x7fef34075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- 172.21.15.89:0/2744100144 shutdown_connections 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- 172.21.15.89:0/2744100144 wait complete. 2023-08-25T02:30:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 Processor -- start 2023-08-25T02:30:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- start start 2023-08-25T02:30:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 0x7fef34197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef341a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341a2e00 con 0x7fef34105840 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341a2f70 con 0x7fef34100440 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.076+0000 7fef3aa6f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341a3130 con 0x7fef340ffa80 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3880b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3900c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef341a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 0x7fef34197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3880b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:41032/0 (socket says 172.21.15.89:41032) 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3880b700 1 -- 172.21.15.89:0/87566091 learned_addr learned my addr 172.21.15.89:0/87566091 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 -- 172.21.15.89:0/87566091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 msgr2=0x7fef34197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 -- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 msgr2=0x7fef341a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef341a2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 -- 172.21.15.89:0/87566091 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280095f0 con 0x7fef34100440 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3880b700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3900c700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef341a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef33fff700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 0x7fef34197ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef2000e9f0 tx=0x7fef2000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef31ffb700 1 -- 172.21.15.89:0/87566091 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000c9c0 con 0x7fef34100440 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef341a3410 con 0x7fef34100440 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef31ffb700 1 -- 172.21.15.89:0/87566091 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef20010430 con 0x7fef34100440 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.077+0000 7fef31ffb700 1 -- 172.21.15.89:0/87566091 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20018680 con 0x7fef34100440 2023-08-25T02:30:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.078+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef341a3970 con 0x7fef34100440 2023-08-25T02:30:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.080+0000 7fef31ffb700 1 -- 172.21.15.89:0/87566091 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fef20010860 con 0x7fef34100440 2023-08-25T02:30:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.080+0000 7fef31ffb700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 0x7fef240a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.080+0000 7fef3880b700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 0x7fef240a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.080+0000 7fef31ffb700 1 -- 172.21.15.89:0/87566091 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fef20014070 con 0x7fef34100440 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3880b700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 0x7fef240a2580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef280036c0 tx=0x7fef28005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 Processor -- start 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 -- start start 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ab450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac713750 con 0x5625ac7ab990 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac704720 con 0x5625ac7b0f40 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef46c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac702040 con 0x5625ac7b1860 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49106/0 (socket says 172.21.15.89:49106) 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 -- 172.21.15.89:0/1072588568 learned_addr learned my addr 172.21.15.89:0/1072588568 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3880b700 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef33fff700 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 -- 172.21.15.89:0/1072588568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 msgr2=0x5625ac7ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 -- 172.21.15.89:0/1072588568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 msgr2=0x5625ac7b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.081+0000 7fef3900c700 1 -- 172.21.15.89:0/1072588568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625ac57d5d0 con 0x5625ac7ab990 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef3900c700 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fef2c003eb0 tx=0x7fef2c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef9dbde0fbd2db7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef1effd700 1 -- 172.21.15.89:0/1072588568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0092d0 con 0x5625ac7ab990 2023-08-25T02:30:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef1effd700 1 -- 172.21.15.89:0/1072588568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef2c00f040 con 0x5625ac7ab990 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef1effd700 1 -- 172.21.15.89:0/1072588568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00e750 con 0x5625ac7ab990 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 -- 172.21.15.89:0/1072588568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 msgr2=0x5625ac7abdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fef2c003eb0 tx=0x7fef2c009920 comp rx=0 tx=0).stop 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 -- 172.21.15.89:0/1072588568 shutdown_connections 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7abdd0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1072588568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.082+0000 7fef46c13b80 1 -- 172.21.15.89:0/1072588568 >> 172.21.15.89:0/1072588568 conn(0x5625ac720080 msgr2=0x5625ac7a5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- 172.21.15.89:0/1072588568 shutdown_connections 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- 172.21.15.89:0/1072588568 wait complete. 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- start start 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 Processor -- start 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b1860 0x5625ac7b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac6fd930 con 0x5625ac7b0f40 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac69bb50 con 0x5625ac7b1860 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac69b850 con 0x5625ac7ab990 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.083+0000 7fef46c13b80 10 client.0 _ll_register_callbacks cb 0x5625ac5e5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef3880b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef3880b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:41046/0 (socket says 172.21.15.89:41046) 2023-08-25T02:30:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef3880b700 1 -- 172.21.15.89:0/2040745314 learned_addr learned my addr 172.21.15.89:0/2040745314 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef3900c700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b1860 0x5625ac7b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 -- 172.21.15.89:0/2040745314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 msgr2=0x5625ac7b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 -- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b1860 msgr2=0x5625ac7b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b1860 0x5625ac7b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 -- 172.21.15.89:0/2040745314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c003b90 con 0x5625ac7b0f40 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef3880b700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef33fff700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7b48b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fef200c8ec0 tx=0x7fef20099580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255a274cee2c6cdf server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef13fff700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef200992e0 con 0x5625ac7b0f40 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fef2009ccb0 con 0x5625ac7b0f40 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625ac77b160 con 0x5625ac7b0f40 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef200a0c90 con 0x5625ac7b0f40 2023-08-25T02:30:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef46c13b80 10 client.15645 fetch_fsmap learned FSMap version 33 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.084+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5625ac529d80 con 0x5625ac7b0f40 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5625ac77b160 con 0x5625ac7b0f40 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 33) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fef20094c70 con 0x5625ac7b0f40 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 10 client.15645 fetch_fsmap finished waiting for FSMap version 33 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 10 client.15645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 20 client.15645 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac7c5100 0x5625ac7c7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 4 client.15645 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625ac6a3b40 con 0x5625ac7c5100 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef3880b700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac7c5100 0x5625ac7c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef13fff700 10 client.15645 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.085+0000 7fef3880b700 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac7c5100 0x5625ac7c7520 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574bcefe06d72f90 server_cookie=ccc4fdee7ee5d4f9 in_seq=0 out_seq=0 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef13fff700 1 -- 172.21.15.89:0/2040745314 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5625ac6a3b40 con 0x5625ac7c5100 2023-08-25T02:30:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef13fff700 10 client.15645 handle_command_reply: tid=0 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 client.15645 shutdown 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 msgr2=0x5625ac7b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7b48b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fef200c8ec0 tx=0x7fef20099580 comp rx=0 tx=0).stop 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 shutdown_connections 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac7c5100 0x5625ac7c7520 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b1860 0x5625ac7b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7ab990 0x5625ac7b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef13fff700 0 client.15645 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2040745314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7b48b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 >> 172.21.15.89:0/2040745314 conn(0x5625ac720080 msgr2=0x5625ac7a0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 shutdown_connections 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 1 -- 172.21.15.89:0/2040745314 wait complete. 2023-08-25T02:30:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.086+0000 7fef46c13b80 20 client.15645 trim_cache size 0 max 16384 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 Processor -- start 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 -- start start 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac57f8c0 con 0x5625ac7ab990 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac7a2560 con 0x5625ac7b0f40 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef46c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac7b59f0 con 0x5625ac7b1860 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.101+0000 7fef3880b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef3880b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49118/0 (socket says 172.21.15.89:49118) 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef3900c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef3880b700 1 -- 172.21.15.89:0/1985348091 learned_addr learned my addr 172.21.15.89:0/1985348091 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 -- 172.21.15.89:0/1985348091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 msgr2=0x5625ac7b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 -- 172.21.15.89:0/1985348091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 msgr2=0x5625ac7ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 -- 172.21.15.89:0/1985348091 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625ac512c00 con 0x5625ac7b0f40 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef3900c700 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef3880b700 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef33fff700 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7bc240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef20098b80 tx=0x7fef200a0b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f625fc1c99b18d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef1f7fe700 1 -- 172.21.15.89:0/1985348091 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2009ccb0 con 0x5625ac7b0f40 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef1f7fe700 1 -- 172.21.15.89:0/1985348091 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef20099e90 con 0x5625ac7b0f40 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.102+0000 7fef1f7fe700 1 -- 172.21.15.89:0/1985348091 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef200985d0 con 0x5625ac7b0f40 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- 172.21.15.89:0/1985348091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 msgr2=0x5625ac7bc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7bc240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef20098b80 tx=0x7fef200a0b50 comp rx=0 tx=0).stop 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- 172.21.15.89:0/1985348091 shutdown_connections 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7ab990 0x5625ac7ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 --2- 172.21.15.89:0/1985348091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7b0f40 0x5625ac7bc240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- 172.21.15.89:0/1985348091 >> 172.21.15.89:0/1985348091 conn(0x5625ac720080 msgr2=0x5625ac7ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- 172.21.15.89:0/1985348091 shutdown_connections 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- 172.21.15.89:0/1985348091 wait complete. 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 -- start start 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.103+0000 7fef46c13b80 1 Processor -- start 2023-08-25T02:30:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac7ad9a0 con 0x5625ac7b0f40 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac7adb10 con 0x5625ac7ab990 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ac7adcd0 con 0x5625ac7b1860 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef46c13b80 10 client.0 _ll_register_callbacks cb 0x7fef20098ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef33fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:39534/0 (socket says 172.21.15.89:39534) 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef33fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49126/0 (socket says 172.21.15.89:49126) 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 -- 172.21.15.89:0/2986880539 learned_addr learned my addr 172.21.15.89:0/2986880539 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3900c700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 -- 172.21.15.89:0/2986880539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 msgr2=0x5625ac7ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 -- 172.21.15.89:0/2986880539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 msgr2=0x5625ac7acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 -- 172.21.15.89:0/2986880539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3410cee0 con 0x5625ac7ab990 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef33fff700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3900c700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef3880b700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c010 tx=0x7fef28009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc24f6334caf5a29 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.104+0000 7fef13fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2023-08-25T02:30:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28029040 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625ac79e600 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fef28033c70 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef280335c0 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 10 client.25250 fetch_fsmap learned FSMap version 33 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5625ac57d5d0 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5625ac79e600 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fef28045d20 con 0x5625ac7ab990 2023-08-25T02:30:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 10 client.25250 fetch_fsmap finished waiting for FSMap version 33 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 10 client.25250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 20 client.25250 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac819c20 0x5625ac81c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 4 client.25250 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5625ac6f6910 con 0x5625ac819c20 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.105+0000 7fef33fff700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac819c20 0x5625ac81c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.106+0000 7fef13fff700 10 client.25250 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:34.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.106+0000 7fef33fff700 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac819c20 0x5625ac81c040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1b1bbb3d070217e server_cookie=2164f2054f0729ff in_seq=0 out_seq=0 2023-08-25T02:30:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef13fff700 1 -- 172.21.15.89:0/2986880539 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5625ac6f6910 con 0x5625ac819c20 2023-08-25T02:30:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef13fff700 10 client.25250 handle_command_reply: tid=0 2023-08-25T02:30:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 client.25250 shutdown 2023-08-25T02:30:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 msgr2=0x5625ac7ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c010 tx=0x7fef28009cd0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 shutdown_connections 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5625ac819c20 0x5625ac81c040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5625ac7ab990 0x5625ac7ac880 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5625ac7b1860 0x5625ac7ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef13fff700 0 client.25250 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 --2- 172.21.15.89:0/2986880539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac7b0f40 0x5625ac7acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 >> 172.21.15.89:0/2986880539 conn(0x5625ac720080 msgr2=0x5625ac7b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 shutdown_connections 2023-08-25T02:30:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 1 -- 172.21.15.89:0/2986880539 wait complete. 2023-08-25T02:30:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.225+0000 7fef46c13b80 20 client.25250 trim_cache size 0 max 16384 2023-08-25T02:30:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:30:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:30:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:30:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:30:34.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 msgr2=0x7fef240a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 0x7fef240a2580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef280036c0 tx=0x7fef28005fb0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 msgr2=0x7fef34197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:34.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 0x7fef34197ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef2000e9f0 tx=0x7fef2000ed00 comp rx=0 tx=0).stop 2023-08-25T02:30:34.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 shutdown_connections 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fef240a00c0 0x7fef240a2580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef34105840 0x7fef341a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef340ffa80 0x7fef34197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 --2- 172.21.15.89:0/87566091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef34100440 0x7fef34197ea0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 >> 172.21.15.89:0/87566091 conn(0x7fef34075250 msgr2=0x7fef340fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 shutdown_connections 2023-08-25T02:30:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:34.226+0000 7fef3aa6f700 1 -- 172.21.15.89:0/87566091 wait complete. 2023-08-25T02:30:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:35 smithi089 ceph-mon[98732]: pgmap v125: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2023-08-25T02:30:35.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:35 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:35 smithi121 ceph-mon[100099]: pgmap v125: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2023-08-25T02:30:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:35 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:35 smithi122 ceph-mon[100457]: pgmap v125: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.4 KiB/s wr, 7 op/s 2023-08-25T02:30:35.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:35 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:30:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:37 smithi089 ceph-mon[98732]: pgmap v126: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-08-25T02:30:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:37 smithi121 ceph-mon[100099]: pgmap v126: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-08-25T02:30:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:37 smithi122 ceph-mon[100457]: pgmap v126: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-08-25T02:30:38.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:38 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:30:38.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:38 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:30:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:38 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:30:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:39 smithi089 ceph-mon[98732]: pgmap v127: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:30:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:30:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:30:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:39 smithi121 ceph-mon[100099]: pgmap v127: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:30:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:30:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:30:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:39 smithi122 ceph-mon[100457]: pgmap v127: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:30:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:30:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:30:41.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:41 smithi121 ceph-mon[100099]: pgmap v128: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:41.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:41 smithi122 ceph-mon[100457]: pgmap v128: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:41 smithi089 ceph-mon[98732]: pgmap v128: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-08-25T02:30:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:43 smithi089 ceph-mon[98732]: pgmap v129: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-08-25T02:30:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:43 smithi121 ceph-mon[100099]: pgmap v129: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-08-25T02:30:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:43 smithi122 ceph-mon[100457]: pgmap v129: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-08-25T02:30:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:45 smithi089 ceph-mon[98732]: pgmap v130: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-08-25T02:30:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:45 smithi121 ceph-mon[100099]: pgmap v130: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-08-25T02:30:45.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:45 smithi122 ceph-mon[100457]: pgmap v130: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-08-25T02:30:47.348 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 -- 172.21.15.89:0/3739902435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7200ffa80 msgr2=0x7fe7200fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 --2- 172.21.15.89:0/3739902435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7200ffa80 0x7fe7200fff00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe7100099c0 tx=0x7fe710009cd0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 -- 172.21.15.89:0/3739902435 shutdown_connections 2023-08-25T02:30:47.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 --2- 172.21.15.89:0/3739902435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720105840 0x7fe720105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 --2- 172.21.15.89:0/3739902435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe720100440 0x7fe72010c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 --2- 172.21.15.89:0/3739902435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7200ffa80 0x7fe7200fff00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 -- 172.21.15.89:0/3739902435 >> 172.21.15.89:0/3739902435 conn(0x7fe720075250 msgr2=0x7fe720075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 -- 172.21.15.89:0/3739902435 shutdown_connections 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.344+0000 7fe725dcf700 1 -- 172.21.15.89:0/3739902435 wait complete. 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 Processor -- start 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 -- start start 2023-08-25T02:30:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720100440 0x7fe720197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 0x7fe7201a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7201a2f20 con 0x7fe720105840 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7201a3090 con 0x7fe720100440 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe725dcf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7201a3250 con 0x7fe7200ffa80 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 0x7fe7201a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 0x7fe7201a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49142/0 (socket says 172.21.15.89:49142) 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:41074/0 (socket says 172.21.15.89:41074) 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.345+0000 7fe71ffff700 1 -- 172.21.15.89:0/186674432 learned_addr learned my addr 172.21.15.89:0/186674432 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71effd700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720100440 0x7fe720197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 -- 172.21.15.89:0/186674432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720100440 msgr2=0x7fe720197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720100440 0x7fe720197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 -- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 msgr2=0x7fe7201a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 0x7fe7201a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 -- 172.21.15.89:0/186674432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100095f0 con 0x7fe7200ffa80 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe71400d750 tx=0x7fe71400da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71cff9700 1 -- 172.21.15.89:0/186674432 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714004220 con 0x7fe7200ffa80 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7201a3530 con 0x7fe7200ffa80 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71cff9700 1 -- 172.21.15.89:0/186674432 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe714004cf0 con 0x7fe7200ffa80 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe71cff9700 1 -- 172.21.15.89:0/186674432 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71400f690 con 0x7fe7200ffa80 2023-08-25T02:30:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.346+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7201a3a10 con 0x7fe7200ffa80 2023-08-25T02:30:47.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.348+0000 7fe71cff9700 1 -- 172.21.15.89:0/186674432 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fe714004750 con 0x7fe7200ffa80 2023-08-25T02:30:47.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.349+0000 7fe71cff9700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 0x7fe7080a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.349+0000 7fe71cff9700 1 -- 172.21.15.89:0/186674432 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe7140cb280 con 0x7fe7200ffa80 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.349+0000 7fe71effd700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 0x7fe7080a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.349+0000 7fe71effd700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 0x7fe7080a2580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe710009990 tx=0x7fe71001a040 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 Processor -- start 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 -- start start 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b73f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b739dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b6a1750 con 0x55701b73f860 2023-08-25T02:30:47.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b692720 con 0x55701b73ef40 2023-08-25T02:30:47.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe731f73b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b690040 con 0x55701b739990 2023-08-25T02:30:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49148/0 (socket says 172.21.15.89:49148) 2023-08-25T02:30:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 -- 172.21.15.89:0/2714934810 learned_addr learned my addr 172.21.15.89:0/2714934810 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71ffff700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b739dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b73f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 -- 172.21.15.89:0/2714934810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 msgr2=0x55701b739dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b739dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 -- 172.21.15.89:0/2714934810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 msgr2=0x55701b73f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b73f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 -- 172.21.15.89:0/2714934810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55701b50b5d0 con 0x55701b73f860 2023-08-25T02:30:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b73f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:30:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71ffff700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b739dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.350+0000 7fe71effd700 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe710005850 tx=0x7fe71001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587331b503bd5ea5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe705ffb700 1 -- 172.21.15.89:0/2714934810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71001d040 con 0x55701b73f860 2023-08-25T02:30:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe705ffb700 1 -- 172.21.15.89:0/2714934810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe710029040 con 0x55701b73f860 2023-08-25T02:30:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe705ffb700 1 -- 172.21.15.89:0/2714934810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710023c10 con 0x55701b73f860 2023-08-25T02:30:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 -- 172.21.15.89:0/2714934810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 msgr2=0x55701b739450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe710005850 tx=0x7fe71001e000 comp rx=0 tx=0).stop 2023-08-25T02:30:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 -- 172.21.15.89:0/2714934810 shutdown_connections 2023-08-25T02:30:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b73f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b739dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2714934810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b739450 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 -- 172.21.15.89:0/2714934810 >> 172.21.15.89:0/2714934810 conn(0x55701b6ae080 msgr2=0x55701b733d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 -- 172.21.15.89:0/2714934810 shutdown_connections 2023-08-25T02:30:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.351+0000 7fe731f73b80 1 -- 172.21.15.89:0/2714934810 wait complete. 2023-08-25T02:30:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 -- start start 2023-08-25T02:30:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 Processor -- start 2023-08-25T02:30:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b742860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b742e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b68b930 con 0x55701b739990 2023-08-25T02:30:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b629b50 con 0x55701b73ef40 2023-08-25T02:30:47.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b629850 con 0x55701b73f860 2023-08-25T02:30:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe731f73b80 10 client.0 _ll_register_callbacks cb 0x55701b573a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49152/0 (socket says 172.21.15.89:49152) 2023-08-25T02:30:47.370 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe71f7fe700 1 -- 172.21.15.89:0/1888778068 learned_addr learned my addr 172.21.15.89:0/1888778068 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.352+0000 7fe71ffff700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b742e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71effd700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b742860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 -- 172.21.15.89:0/1888778068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 msgr2=0x55701b742e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b742e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 -- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 msgr2=0x55701b742860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b742860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 -- 172.21.15.89:0/1888778068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71800e330 con 0x55701b739990 2023-08-25T02:30:47.372 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71effd700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b742860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:30:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe714098920 tx=0x7fe7140982f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3629ec191befc515 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:30:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140057a0 con 0x55701b739990 2023-08-25T02:30:47.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe714083040 con 0x55701b739990 2023-08-25T02:30:47.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55701b709160 con 0x55701b739990 2023-08-25T02:30:47.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe714098990 con 0x55701b739990 2023-08-25T02:30:47.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 10 client.15663 fetch_fsmap learned FSMap version 34 2023-08-25T02:30:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7fe72010cee0 con 0x55701b739990 2023-08-25T02:30:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55701b709160 con 0x55701b739990 2023-08-25T02:30:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 34) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe7140b6020 con 0x55701b739990 2023-08-25T02:30:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 10 client.15663 fetch_fsmap finished waiting for FSMap version 34 2023-08-25T02:30:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 20 client.15663 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b752fd0 0x55701b7553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.353+0000 7fe731f73b80 4 client.15663 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:30:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.354+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55701b631b40 con 0x55701b752fd0 2023-08-25T02:30:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.354+0000 7fe71effd700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b752fd0 0x55701b7553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.354+0000 7fe6feffd700 10 client.15663 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.354+0000 7fe71effd700 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b752fd0 0x55701b7553c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb269842d946d96 server_cookie=3cb2d0038c22ad12 in_seq=0 out_seq=0 2023-08-25T02:30:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe6feffd700 1 -- 172.21.15.89:0/1888778068 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55701b631b40 con 0x55701b752fd0 2023-08-25T02:30:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe6feffd700 10 client.15663 handle_command_reply: tid=0 2023-08-25T02:30:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 client.15663 shutdown 2023-08-25T02:30:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 msgr2=0x55701b745650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe714098920 tx=0x7fe7140982f0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 shutdown_connections 2023-08-25T02:30:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b752fd0 0x55701b7553c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b742e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe6feffd700 0 client.15663 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b739990 0x55701b745650 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 --2- 172.21.15.89:0/1888778068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b742860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 >> 172.21.15.89:0/1888778068 conn(0x55701b6ae080 msgr2=0x55701b72e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 shutdown_connections 2023-08-25T02:30:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 1 -- 172.21.15.89:0/1888778068 wait complete. 2023-08-25T02:30:47.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.355+0000 7fe731f73b80 20 client.15663 trim_cache size 0 max 16384 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.369+0000 7fe731f73b80 1 Processor -- start 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 -- start start 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b745180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b50d8c0 con 0x55701b73f860 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b74a750 con 0x55701b73ef40 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe731f73b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b740de0 con 0x55701b739990 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:41096/0 (socket says 172.21.15.89:41096) 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49164/0 (socket says 172.21.15.89:49164) 2023-08-25T02:30:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 -- 172.21.15.89:0/2297805692 learned_addr learned my addr 172.21.15.89:0/2297805692 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71effd700 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b745180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 -- 172.21.15.89:0/2297805692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 msgr2=0x55701b745180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b745180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 -- 172.21.15.89:0/2297805692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 msgr2=0x55701b73fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 -- 172.21.15.89:0/2297805692 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55701b4b7d80 con 0x55701b739990 2023-08-25T02:30:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71ffff700 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe714095400 tx=0x7fe714096f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f046aa799dad3c38 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.370+0000 7fe7067fc700 1 -- 172.21.15.89:0/2297805692 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714014070 con 0x55701b739990 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe7067fc700 1 -- 172.21.15.89:0/2297805692 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe714097f70 con 0x55701b739990 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe7067fc700 1 -- 172.21.15.89:0/2297805692 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140a0550 con 0x55701b739990 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- 172.21.15.89:0/2297805692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 msgr2=0x55701b74a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe714095400 tx=0x7fe714096f40 comp rx=0 tx=0).stop 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- 172.21.15.89:0/2297805692 shutdown_connections 2023-08-25T02:30:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73f860 0x55701b73fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b739990 0x55701b74a210 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 --2- 172.21.15.89:0/2297805692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b73ef40 0x55701b745180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- 172.21.15.89:0/2297805692 >> 172.21.15.89:0/2297805692 conn(0x55701b6ae080 msgr2=0x55701b733bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- 172.21.15.89:0/2297805692 shutdown_connections 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- 172.21.15.89:0/2297805692 wait complete. 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.371+0000 7fe731f73b80 1 -- start start 2023-08-25T02:30:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 Processor -- start 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b739990 0x55701b73b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b73a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b73c650 con 0x55701b73ef40 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b73adc0 con 0x55701b739990 2023-08-25T02:30:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701b73af50 con 0x55701b73f860 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe731f73b80 10 client.0 _ll_register_callbacks cb 0x7fe7140987a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49168/0 (socket says 172.21.15.89:49168) 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 -- 172.21.15.89:0/3723802443 learned_addr learned my addr 172.21.15.89:0/3723802443 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71ffff700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b73a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b739990 0x55701b73b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 -- 172.21.15.89:0/3723802443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 msgr2=0x55701b73a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b73a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 -- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b739990 msgr2=0x55701b73b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b739990 0x55701b73b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71effd700 1 -- 172.21.15.89:0/3723802443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71800e680 con 0x55701b73ef40 2023-08-25T02:30:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.372+0000 7fe71ffff700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b73a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe71effd700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe71000a010 tx=0x7fe710005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988fe2e9bab7bdad server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71001d040 con 0x55701b73ef40 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fe71000fc90 con 0x55701b73ef40 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55701b72c600 con 0x55701b73ef40 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe71000fe50 con 0x55701b73ef40 2023-08-25T02:30:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 10 client.15669 fetch_fsmap learned FSMap version 34 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7fe72010cee0 con 0x55701b73ef40 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55701b72c600 con 0x55701b73ef40 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 34) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe71002c070 con 0x55701b73ef40 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 10 client.15669 fetch_fsmap finished waiting for FSMap version 34 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 10 client.15669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 20 client.15669 populate_metadata read hostname 'smithi089' 2023-08-25T02:30:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b7abb50 0x55701b7adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 4 client.15669 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.373+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55701b684910 con 0x55701b7abb50 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.374+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b7abb50 0x55701b7adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.374+0000 7fe6feffd700 10 client.15669 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.374+0000 7fe71f7fe700 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b7abb50 0x55701b7adf70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbfbac7e20076615 server_cookie=a86a0f6b748f3184 in_seq=0 out_seq=0 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe6feffd700 1 -- 172.21.15.89:0/3723802443 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55701b684910 con 0x55701b7abb50 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe6feffd700 10 client.15669 handle_command_reply: tid=0 2023-08-25T02:30:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 client.15669 shutdown 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 msgr2=0x55701b73be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe71000a010 tx=0x7fe710005c80 comp rx=0 tx=0).stop 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 shutdown_connections 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55701b7abb50 0x55701b7adf70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55701b739990 0x55701b73b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55701b73f860 0x55701b73a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 --2- 172.21.15.89:0/3723802443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55701b73ef40 0x55701b73be40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe6feffd700 0 client.15669 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 >> 172.21.15.89:0/3723802443 conn(0x55701b6ae080 msgr2=0x55701b72df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 shutdown_connections 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 1 -- 172.21.15.89:0/3723802443 wait complete. 2023-08-25T02:30:47.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe731f73b80 20 client.15669 trim_cache size 0 max 16384 2023-08-25T02:30:47.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:30:47.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:30:47.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:30:47.401 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:30:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 msgr2=0x7fe7080a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 0x7fe7080a2580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe710009990 tx=0x7fe71001a040 comp rx=0 tx=0).stop 2023-08-25T02:30:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 msgr2=0x7fe720197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe71400d750 tx=0x7fe71400da60 comp rx=0 tx=0).stop 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 shutdown_connections 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fe7080a00c0 0x7fe7080a2580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe720105840 0x7fe7201a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7200ffa80 0x7fe720197ab0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 --2- 172.21.15.89:0/186674432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720100440 0x7fe720197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.381+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 >> 172.21.15.89:0/186674432 conn(0x7fe720075250 msgr2=0x7fe7200fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:30:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.382+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 shutdown_connections 2023-08-25T02:30:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:30:47.382+0000 7fe725dcf700 1 -- 172.21.15.89:0/186674432 wait complete. 2023-08-25T02:30:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:47 smithi089 ceph-mon[98732]: pgmap v131: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 KiB/s wr, 11 op/s 2023-08-25T02:30:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:47 smithi121 ceph-mon[100099]: pgmap v131: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 KiB/s wr, 11 op/s 2023-08-25T02:30:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:47 smithi122 ceph-mon[100457]: pgmap v131: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 14 KiB/s wr, 11 op/s 2023-08-25T02:30:49.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:49 smithi089 ceph-mon[98732]: pgmap v132: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:49 smithi121 ceph-mon[100099]: pgmap v132: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:49 smithi122 ceph-mon[100457]: pgmap v132: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:51.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:51 smithi121 ceph-mon[100099]: pgmap v133: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:51 smithi122 ceph-mon[100457]: pgmap v133: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:51 smithi089 ceph-mon[98732]: pgmap v133: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2023-08-25T02:30:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:53 smithi089 ceph-mon[98732]: pgmap v134: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-08-25T02:30:53.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:53 smithi121 ceph-mon[100099]: pgmap v134: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-08-25T02:30:53.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:53 smithi122 ceph-mon[100457]: pgmap v134: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-08-25T02:30:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:55 smithi089 ceph-mon[98732]: pgmap v135: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 16 KiB/s wr, 10 op/s 2023-08-25T02:30:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:55 smithi121 ceph-mon[100099]: pgmap v135: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 16 KiB/s wr, 10 op/s 2023-08-25T02:30:55.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:55 smithi122 ceph-mon[100457]: pgmap v135: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 16 KiB/s wr, 10 op/s 2023-08-25T02:30:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:57 smithi089 ceph-mon[98732]: pgmap v136: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 KiB/s wr, 14 op/s 2023-08-25T02:30:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:57 smithi121 ceph-mon[100099]: pgmap v136: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 KiB/s wr, 14 op/s 2023-08-25T02:30:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:57 smithi122 ceph-mon[100457]: pgmap v136: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 KiB/s wr, 14 op/s 2023-08-25T02:30:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: pgmap v137: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: 180 slow requests, 5 included below; oldest blocked for > 183.031670 secs 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: slow request 182.772107 seconds old, received at 2023-08-25T02:27:55.880420+0000: client_request(mds.1:1387 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: slow request 182.739203 seconds old, received at 2023-08-25T02:27:55.913325+0000: client_request(mds.1:1419 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: slow request 182.707790 seconds old, received at 2023-08-25T02:27:55.944737+0000: client_request(mds.1:1451 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: slow request 182.674481 seconds old, received at 2023-08-25T02:27:55.978046+0000: client_request(mds.1:1483 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:30:59 smithi089 ceph-mon[98732]: slow request 182.641120 seconds old, received at 2023-08-25T02:27:56.011408+0000: client_request(mds.1:1515 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: pgmap v137: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: 180 slow requests, 5 included below; oldest blocked for > 183.031670 secs 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: slow request 182.772107 seconds old, received at 2023-08-25T02:27:55.880420+0000: client_request(mds.1:1387 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: slow request 182.739203 seconds old, received at 2023-08-25T02:27:55.913325+0000: client_request(mds.1:1419 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: slow request 182.707790 seconds old, received at 2023-08-25T02:27:55.944737+0000: client_request(mds.1:1451 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: slow request 182.674481 seconds old, received at 2023-08-25T02:27:55.978046+0000: client_request(mds.1:1483 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:30:59 smithi121 ceph-mon[100099]: slow request 182.641120 seconds old, received at 2023-08-25T02:27:56.011408+0000: client_request(mds.1:1515 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: pgmap v137: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: 180 slow requests, 5 included below; oldest blocked for > 183.031670 secs 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: slow request 182.772107 seconds old, received at 2023-08-25T02:27:55.880420+0000: client_request(mds.1:1387 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: slow request 182.739203 seconds old, received at 2023-08-25T02:27:55.913325+0000: client_request(mds.1:1419 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: slow request 182.707790 seconds old, received at 2023-08-25T02:27:55.944737+0000: client_request(mds.1:1451 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: slow request 182.674481 seconds old, received at 2023-08-25T02:27:55.978046+0000: client_request(mds.1:1483 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:30:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:30:59 smithi122 ceph-mon[100457]: slow request 182.641120 seconds old, received at 2023-08-25T02:27:56.011408+0000: client_request(mds.1:1515 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:31:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 -- 172.21.15.89:0/1000055264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41008a0 msgr2=0x7f0ad4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/1000055264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41008a0 0x7f0ad4100d20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009990 tx=0x7f0ac4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 -- 172.21.15.89:0/1000055264 shutdown_connections 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/1000055264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41068b0 0x7f0ad4106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/1000055264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 0x7f0ad410d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/1000055264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41008a0 0x7f0ad4100d20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 -- 172.21.15.89:0/1000055264 >> 172.21.15.89:0/1000055264 conn(0x7f0ad40fc370 msgr2=0x7f0ad40fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 -- 172.21.15.89:0/1000055264 shutdown_connections 2023-08-25T02:31:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.501+0000 7f0ad8a13700 1 -- 172.21.15.89:0/1000055264 wait complete. 2023-08-25T02:31:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 Processor -- start 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 -- start start 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41008a0 0x7f0ad4072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 0x7f0ad4073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41a5060 con 0x7f0ad41068b0 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41a51a0 con 0x7f0ad41008a0 2023-08-25T02:31:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad8a13700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41a5310 con 0x7f0ad4101260 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 0x7f0ad4073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49184/0 (socket says 172.21.15.89:49184) 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/137812934 learned_addr learned my addr 172.21.15.89:0/137812934 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.502+0000 7f0ad259c700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41008a0 0x7f0ad4072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/137812934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 msgr2=0x7f0ad4073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 0x7f0ad4073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/137812934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41008a0 msgr2=0x7f0ad4072e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41008a0 0x7f0ad4072e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/137812934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac40095f0 con 0x7f0ad41068b0 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00ef30 tx=0x7f0acc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ac37fe700 1 -- 172.21.15.89:0/137812934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc018070 con 0x7f0ad41068b0 2023-08-25T02:31:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ac37fe700 1 -- 172.21.15.89:0/137812934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0acc00f040 con 0x7f0ad41068b0 2023-08-25T02:31:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad41a5590 con 0x7f0ad41068b0 2023-08-25T02:31:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ac37fe700 1 -- 172.21.15.89:0/137812934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc00cd00 con 0x7f0ad41068b0 2023-08-25T02:31:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.503+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41a5aa0 con 0x7f0ad41068b0 2023-08-25T02:31:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.505+0000 7f0ac37fe700 1 -- 172.21.15.89:0/137812934 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f0acc003a00 con 0x7f0ad41068b0 2023-08-25T02:31:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.506+0000 7f0ac37fe700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 0x7f0abc0a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.506+0000 7f0ad259c700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 0x7f0abc0a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.506+0000 7f0ac37fe700 1 -- 172.21.15.89:0/137812934 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f0acc0cb140 con 0x7f0ad41068b0 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.506+0000 7f0ad259c700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 0x7f0abc0a2600 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009be0 tx=0x7f0ac8009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 Processor -- start 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 -- start start 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 0x559f21857320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 0x559f21851dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f217b9750 con 0x559f21857860 2023-08-25T02:31:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f217aa720 con 0x559f21851990 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f217a8040 con 0x559f21856f40 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49190/0 (socket says 172.21.15.89:49190) 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ad259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 0x559f21857320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ad1d9b700 1 -- 172.21.15.89:0/180098360 learned_addr learned my addr 172.21.15.89:0/180098360 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.507+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 0x559f21851dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 -- 172.21.15.89:0/180098360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 msgr2=0x559f21851dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 0x559f21851dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 -- 172.21.15.89:0/180098360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 msgr2=0x559f21851450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 -- 172.21.15.89:0/180098360 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f216235d0 con 0x559f21856f40 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 0x559f21851dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad1d9b700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.508+0000 7f0ad259c700 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 0x559f21857320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80117d0 tx=0x7f0ac8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7244d5f0c161e81f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.509+0000 7f0ac0ff9700 1 -- 172.21.15.89:0/180098360 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80119e0 con 0x559f21856f40 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.509+0000 7f0ac0ff9700 1 -- 172.21.15.89:0/180098360 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac80044e0 con 0x559f21856f40 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.509+0000 7f0ac0ff9700 1 -- 172.21.15.89:0/180098360 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8003aa0 con 0x559f21856f40 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/180098360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 msgr2=0x559f21857320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 0x559f21857320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80117d0 tx=0x7f0ac8014000 comp rx=0 tx=0).stop 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/180098360 shutdown_connections 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21851990 0x559f21851dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21856f40 0x559f21857320 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/180098360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21857860 0x559f21851450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/180098360 >> 172.21.15.89:0/180098360 conn(0x559f217c6080 msgr2=0x559f2184bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/180098360 shutdown_connections 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/180098360 wait complete. 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 -- start start 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.510+0000 7f0ae4bb7b80 1 Processor -- start 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f2185a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 0x559f2185aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f217a3930 con 0x559f21851990 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f21741b50 con 0x559f21856f40 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f21741850 con 0x559f21857860 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ae4bb7b80 10 client.0 _ll_register_callbacks cb 0x559f2168ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f2185a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 0x559f2185aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49194/0 (socket says 172.21.15.89:49194) 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad259c700 1 -- 172.21.15.89:0/3867543214 learned_addr learned my addr 172.21.15.89:0/3867543214 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 msgr2=0x559f2185a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f2185a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/3867543214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 msgr2=0x559f2185d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/3867543214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad410e070 con 0x559f21857860 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 0x559f2185aec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0c8890 tx=0x7f0acc0982c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=915c6a1bcf0560be server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ab9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.511+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc098fe0 con 0x559f21857860 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f0acc00ce60 con 0x559f21857860 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f21821160 con 0x559f21857860 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0acc0988e0 con 0x559f21857860 2023-08-25T02:31:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 10 client.25303 fetch_fsmap learned FSMap version 35 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f0ac8017070 con 0x559f21857860 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0acc0c6080 con 0x559f21857860 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 35) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0acc0a01d0 con 0x559f21857860 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 10 client.25303 fetch_fsmap finished waiting for FSMap version 35 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 10 client.25303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 20 client.25303 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f2186b000 0x559f2186d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 4 client.25303 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:31:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f21749b40 con 0x559f2186b000 2023-08-25T02:31:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.512+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f2186b000 0x559f2186d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.513+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f2186b000 0x559f2186d420 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b2ae8df6fbd320 server_cookie=1c3bb625377835f4 in_seq=0 out_seq=0 2023-08-25T02:31:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.513+0000 7f0ab9ffb700 10 client.25303 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.513+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3867543214 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559f21749b40 con 0x559f2186b000 2023-08-25T02:31:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.513+0000 7f0ab9ffb700 10 client.25303 handle_command_reply: tid=0 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.513+0000 7f0ae4bb7b80 1 client.25303 shutdown 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 msgr2=0x559f2185aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 0x559f2185aec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0c8890 tx=0x7f0acc0982c0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 shutdown_connections 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f2186b000 0x559f2186d420 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21857860 0x559f2185aec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ab9ffb700 0 client.25303 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f2185d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3867543214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f2185a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 >> 172.21.15.89:0/3867543214 conn(0x559f217c6080 msgr2=0x559f21846820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 shutdown_connections 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3867543214 wait complete. 2023-08-25T02:31:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.514+0000 7f0ae4bb7b80 20 client.25303 trim_cache size 0 max 16384 2023-08-25T02:31:00.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 Processor -- start 2023-08-25T02:31:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 -- start start 2023-08-25T02:31:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 0x559f218622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21851990 0x559f2185a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f216258c0 con 0x559f21856f40 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f2184c250 con 0x559f217c6080 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f218517a0 con 0x559f21851990 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49206/0 (socket says 172.21.15.89:49206) 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/109239010 learned_addr learned my addr 172.21.15.89:0/109239010 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:00.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad1d9b700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21851990 0x559f2185a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad259c700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 0x559f218622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/109239010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21851990 msgr2=0x559f2185a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21851990 0x559f2185a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/109239010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 msgr2=0x559f218622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 0x559f218622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 -- 172.21.15.89:0/109239010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f215cfd80 con 0x559f21856f40 2023-08-25T02:31:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad259c700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 0x559f218622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:31:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00ef00 tx=0x7f0acc0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9122bb983a0b625a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ac17fa700 1 -- 172.21.15.89:0/109239010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc00ce60 con 0x559f21856f40 2023-08-25T02:31:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.529+0000 7f0ac17fa700 1 -- 172.21.15.89:0/109239010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559f21821160 con 0x559f21856f40 2023-08-25T02:31:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ac17fa700 1 -- 172.21.15.89:0/109239010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a06d0 con 0x559f21856f40 2023-08-25T02:31:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/109239010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 msgr2=0x559f2185d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00ef00 tx=0x7f0acc0092e0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/109239010 shutdown_connections 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f217c6080 0x559f218622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21856f40 0x559f2185d190 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/109239010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f21851990 0x559f2185a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/109239010 >> 172.21.15.89:0/109239010 conn(0x559f21852080 msgr2=0x559f21854af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/109239010 shutdown_connections 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/109239010 wait complete. 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 -- start start 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.530+0000 7f0ae4bb7b80 1 Processor -- start 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f21854140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f21852880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f21852df0 con 0x559f21851990 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f21852f60 con 0x559f21856f40 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f21853120 con 0x559f217c6080 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ae4bb7b80 10 client.0 _ll_register_callbacks cb 0x559f2168ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f21854140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.89:41144/0 (socket says 172.21.15.89:41144) 2023-08-25T02:31:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad259c700 1 -- 172.21.15.89:0/3370024289 learned_addr learned my addr 172.21.15.89:0/3370024289 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad2d9d700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f21852880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 -- 172.21.15.89:0/3370024289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 msgr2=0x559f21853c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 -- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 msgr2=0x559f21852880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f21852880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 -- 172.21.15.89:0/3370024289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac8017070 con 0x559f21851990 2023-08-25T02:31:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ad1d9b700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f21854140 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4000c00 tx=0x7f0ac4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b68950e9083381c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.531+0000 7f0ab9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac401d040 con 0x559f21851990 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f0ac4005680 con 0x559f21851990 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f21844600 con 0x559f21851990 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ac4004500 con 0x559f21851990 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 10 client.15687 fetch_fsmap learned FSMap version 35 2023-08-25T02:31:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559f215b8c00 con 0x559f21851990 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559f21844600 con 0x559f21851990 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 35) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0ac402c020 con 0x559f21851990 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 10 client.15687 fetch_fsmap finished waiting for FSMap version 35 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 10 client.15687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 20 client.15687 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f21857860 0x559f218c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 4 client.15687 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:31:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559f2179c910 con 0x559f21857860 2023-08-25T02:31:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.532+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f21857860 0x559f218c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.533+0000 7f0ab9ffb700 10 client.15687 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.533+0000 7f0ad259c700 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f21857860 0x559f218c1c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b74c82b123016505 server_cookie=177e3a5062546685 in_seq=0 out_seq=0 2023-08-25T02:31:00.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ab9ffb700 1 -- 172.21.15.89:0/3370024289 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559f2179c910 con 0x559f21857860 2023-08-25T02:31:00.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ab9ffb700 10 client.15687 handle_command_reply: tid=0 2023-08-25T02:31:00.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 client.15687 shutdown 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 msgr2=0x559f21854140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f21854140 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4000c00 tx=0x7f0ac4005040 comp rx=0 tx=0).stop 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 shutdown_connections 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f21857860 0x559f218c1c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f21856f40 0x559f21852880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f217c6080 0x559f21853c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 --2- 172.21.15.89:0/3370024289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21851990 0x559f21854140 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.541+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 >> 172.21.15.89:0/3370024289 conn(0x559f21852080 msgr2=0x559f21857c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ab9ffb700 0 client.15687 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 shutdown_connections 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ae4bb7b80 1 -- 172.21.15.89:0/3370024289 wait complete. 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ae4bb7b80 20 client.15687 trim_cache size 0 max 16384 2023-08-25T02:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 msgr2=0x7f0abc0a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 0x7f0abc0a2600 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009be0 tx=0x7f0ac8009260 comp rx=0 tx=0).stop 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 msgr2=0x7f0ad41a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00ef30 tx=0x7f0acc00c3a0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 shutdown_connections 2023-08-25T02:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0abc0a0140 0x7f0abc0a2600 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ad41008a0 0x7f0ad4072e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ad41068b0 0x7f0ad41a4b20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 --2- 172.21.15.89:0/137812934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ad4101260 0x7f0ad4073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 >> 172.21.15.89:0/137812934 conn(0x7f0ad40fc370 msgr2=0x7f0ad40fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 shutdown_connections 2023-08-25T02:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:00.542+0000 7f0ad8a13700 1 -- 172.21.15.89:0/137812934 wait complete. 2023-08-25T02:31:01.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:01 smithi121 ceph-mon[100099]: pgmap v138: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:31:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:01 smithi121 ceph-mon[100099]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:01 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:01.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:01 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:01.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:01 smithi122 ceph-mon[100457]: pgmap v138: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:31:01.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:01 smithi122 ceph-mon[100457]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:01.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:01 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:01.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:01 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:01 smithi089 ceph-mon[98732]: pgmap v138: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-08-25T02:31:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:01 smithi089 ceph-mon[98732]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:01 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:01.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:01 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:03 smithi089 ceph-mon[98732]: pgmap v139: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-08-25T02:31:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:03 smithi121 ceph-mon[100099]: pgmap v139: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-08-25T02:31:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:03 smithi122 ceph-mon[100457]: pgmap v139: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-08-25T02:31:04.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: 180 slow requests, 5 included below; oldest blocked for > 188.154312 secs 2023-08-25T02:31:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: slow request 187.730957 seconds old, received at 2023-08-25T02:27:56.044213+0000: client_request(mds.1:1547 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: slow request 187.698626 seconds old, received at 2023-08-25T02:27:56.076543+0000: client_request(mds.1:1579 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: slow request 187.665630 seconds old, received at 2023-08-25T02:27:56.109539+0000: client_request(mds.1:1611 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: slow request 187.632040 seconds old, received at 2023-08-25T02:27:56.143130+0000: client_request(mds.1:1643 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:04 smithi089 ceph-mon[98732]: slow request 187.598415 seconds old, received at 2023-08-25T02:27:56.176755+0000: client_request(mds.1:1675 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: 180 slow requests, 5 included below; oldest blocked for > 188.154312 secs 2023-08-25T02:31:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: slow request 187.730957 seconds old, received at 2023-08-25T02:27:56.044213+0000: client_request(mds.1:1547 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: slow request 187.698626 seconds old, received at 2023-08-25T02:27:56.076543+0000: client_request(mds.1:1579 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: slow request 187.665630 seconds old, received at 2023-08-25T02:27:56.109539+0000: client_request(mds.1:1611 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: slow request 187.632040 seconds old, received at 2023-08-25T02:27:56.143130+0000: client_request(mds.1:1643 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:04 smithi121 ceph-mon[100099]: slow request 187.598415 seconds old, received at 2023-08-25T02:27:56.176755+0000: client_request(mds.1:1675 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: 180 slow requests, 5 included below; oldest blocked for > 188.154312 secs 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: slow request 187.730957 seconds old, received at 2023-08-25T02:27:56.044213+0000: client_request(mds.1:1547 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: slow request 187.698626 seconds old, received at 2023-08-25T02:27:56.076543+0000: client_request(mds.1:1579 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: slow request 187.665630 seconds old, received at 2023-08-25T02:27:56.109539+0000: client_request(mds.1:1611 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: slow request 187.632040 seconds old, received at 2023-08-25T02:27:56.143130+0000: client_request(mds.1:1643 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:04 smithi122 ceph-mon[100457]: slow request 187.598415 seconds old, received at 2023-08-25T02:27:56.176755+0000: client_request(mds.1:1675 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:05 smithi089 ceph-mon[98732]: pgmap v140: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-08-25T02:31:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:05 smithi121 ceph-mon[100099]: pgmap v140: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-08-25T02:31:05.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:05 smithi122 ceph-mon[100457]: pgmap v140: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-08-25T02:31:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:07 smithi089 ceph-mon[98732]: pgmap v141: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 23 op/s 2023-08-25T02:31:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:07 smithi121 ceph-mon[100099]: pgmap v141: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 23 op/s 2023-08-25T02:31:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:07 smithi122 ceph-mon[100457]: pgmap v141: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 23 op/s 2023-08-25T02:31:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: pgmap v142: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: 59 slow requests, 5 included below; oldest blocked for > 183.561598 secs 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: slow request 183.496776 seconds old, received at 2023-08-25T02:28:05.895128+0000: peer_request:mds.1:1397 currently dispatched 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: slow request 183.489065 seconds old, received at 2023-08-25T02:28:05.902839+0000: peer_request:mds.1:1429 currently dispatched 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: slow request 183.481334 seconds old, received at 2023-08-25T02:28:05.910569+0000: peer_request:mds.1:1461 currently dispatched 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: slow request 183.473650 seconds old, received at 2023-08-25T02:28:05.918254+0000: peer_request:mds.1:1493 currently dispatched 2023-08-25T02:31:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:09 smithi089 ceph-mon[98732]: slow request 183.465959 seconds old, received at 2023-08-25T02:28:05.925944+0000: peer_request:mds.1:1525 currently dispatched 2023-08-25T02:31:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: pgmap v142: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: 59 slow requests, 5 included below; oldest blocked for > 183.561598 secs 2023-08-25T02:31:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: slow request 183.496776 seconds old, received at 2023-08-25T02:28:05.895128+0000: peer_request:mds.1:1397 currently dispatched 2023-08-25T02:31:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: slow request 183.489065 seconds old, received at 2023-08-25T02:28:05.902839+0000: peer_request:mds.1:1429 currently dispatched 2023-08-25T02:31:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: slow request 183.481334 seconds old, received at 2023-08-25T02:28:05.910569+0000: peer_request:mds.1:1461 currently dispatched 2023-08-25T02:31:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: slow request 183.473650 seconds old, received at 2023-08-25T02:28:05.918254+0000: peer_request:mds.1:1493 currently dispatched 2023-08-25T02:31:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:09 smithi121 ceph-mon[100099]: slow request 183.465959 seconds old, received at 2023-08-25T02:28:05.925944+0000: peer_request:mds.1:1525 currently dispatched 2023-08-25T02:31:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: pgmap v142: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: 59 slow requests, 5 included below; oldest blocked for > 183.561598 secs 2023-08-25T02:31:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: slow request 183.496776 seconds old, received at 2023-08-25T02:28:05.895128+0000: peer_request:mds.1:1397 currently dispatched 2023-08-25T02:31:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: slow request 183.489065 seconds old, received at 2023-08-25T02:28:05.902839+0000: peer_request:mds.1:1429 currently dispatched 2023-08-25T02:31:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: slow request 183.481334 seconds old, received at 2023-08-25T02:28:05.910569+0000: peer_request:mds.1:1461 currently dispatched 2023-08-25T02:31:09.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: slow request 183.473650 seconds old, received at 2023-08-25T02:28:05.918254+0000: peer_request:mds.1:1493 currently dispatched 2023-08-25T02:31:09.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:09 smithi122 ceph-mon[100457]: slow request 183.465959 seconds old, received at 2023-08-25T02:28:05.925944+0000: peer_request:mds.1:1525 currently dispatched 2023-08-25T02:31:10.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: 299 slow requests, 5 included below; oldest blocked for > 193.337812 secs 2023-08-25T02:31:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: slow request 192.748468 seconds old, received at 2023-08-25T02:27:56.210202+0000: client_request(mds.1:1707 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: slow request 192.714959 seconds old, received at 2023-08-25T02:27:56.243710+0000: client_request(mds.1:1739 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: slow request 192.682171 seconds old, received at 2023-08-25T02:27:56.276499+0000: client_request(mds.1:1771 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: slow request 192.649360 seconds old, received at 2023-08-25T02:27:56.309309+0000: client_request(mds.1:1803 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:10 smithi089 ceph-mon[98732]: slow request 192.616056 seconds old, received at 2023-08-25T02:27:56.342613+0000: client_request(mds.1:1835 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: 299 slow requests, 5 included below; oldest blocked for > 193.337812 secs 2023-08-25T02:31:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: slow request 192.748468 seconds old, received at 2023-08-25T02:27:56.210202+0000: client_request(mds.1:1707 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: slow request 192.714959 seconds old, received at 2023-08-25T02:27:56.243710+0000: client_request(mds.1:1739 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: slow request 192.682171 seconds old, received at 2023-08-25T02:27:56.276499+0000: client_request(mds.1:1771 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: slow request 192.649360 seconds old, received at 2023-08-25T02:27:56.309309+0000: client_request(mds.1:1803 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:10 smithi121 ceph-mon[100099]: slow request 192.616056 seconds old, received at 2023-08-25T02:27:56.342613+0000: client_request(mds.1:1835 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: 299 slow requests, 5 included below; oldest blocked for > 193.337812 secs 2023-08-25T02:31:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: slow request 192.748468 seconds old, received at 2023-08-25T02:27:56.210202+0000: client_request(mds.1:1707 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: slow request 192.714959 seconds old, received at 2023-08-25T02:27:56.243710+0000: client_request(mds.1:1739 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: slow request 192.682171 seconds old, received at 2023-08-25T02:27:56.276499+0000: client_request(mds.1:1771 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: slow request 192.649360 seconds old, received at 2023-08-25T02:27:56.309309+0000: client_request(mds.1:1803 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:10 smithi122 ceph-mon[100457]: slow request 192.616056 seconds old, received at 2023-08-25T02:27:56.342613+0000: client_request(mds.1:1835 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:11 smithi089 ceph-mon[98732]: pgmap v143: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:11 smithi121 ceph-mon[100099]: pgmap v143: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:11 smithi122 ceph-mon[100457]: pgmap v143: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 19 op/s 2023-08-25T02:31:13.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/1142423808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 msgr2=0x7fbd480ffc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/1142423808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd480ffc50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009990 tx=0x7fbd3c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/1142423808 shutdown_connections 2023-08-25T02:31:13.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/1142423808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/1142423808 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 0x7fbd48105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/1142423808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd480ffc50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.658+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/1142423808 >> 172.21.15.89:0/1142423808 conn(0x7fbd480fb260 msgr2=0x7fbd480fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.659+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/1142423808 shutdown_connections 2023-08-25T02:31:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.659+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/1142423808 wait complete. 2023-08-25T02:31:13.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.659+0000 7fbd4dc0c700 1 Processor -- start 2023-08-25T02:31:13.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.659+0000 7fbd4dc0c700 1 -- start start 2023-08-25T02:31:13.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4819bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 0x7fbd481956a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48195d10 con 0x7fbd480ff7d0 2023-08-25T02:31:13.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48195e80 con 0x7fbd48100190 2023-08-25T02:31:13.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48196040 con 0x7fbd481057e0 2023-08-25T02:31:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49220/0 (socket says 172.21.15.89:49220) 2023-08-25T02:31:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 -- 172.21.15.89:0/4280976113 learned_addr learned my addr 172.21.15.89:0/4280976113 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd47fff700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 0x7fbd481956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.669 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4819bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 -- 172.21.15.89:0/4280976113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 msgr2=0x7fbd481956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 0x7fbd481956a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 -- 172.21.15.89:0/4280976113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 msgr2=0x7fbd4819bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4819bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 -- 172.21.15.89:0/4280976113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c0095f0 con 0x7fbd480ff7d0 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4819bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:31:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd477fe700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e9a0 tx=0x7fbd3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd44ff9700 1 -- 172.21.15.89:0/4280976113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38009790 con 0x7fbd480ff7d0 2023-08-25T02:31:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd48196320 con 0x7fbd480ff7d0 2023-08-25T02:31:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.660+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd481a2830 con 0x7fbd480ff7d0 2023-08-25T02:31:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.661+0000 7fbd44ff9700 1 -- 172.21.15.89:0/4280976113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd3800cbd0 con 0x7fbd480ff7d0 2023-08-25T02:31:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.661+0000 7fbd44ff9700 1 -- 172.21.15.89:0/4280976113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38018680 con 0x7fbd480ff7d0 2023-08-25T02:31:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.663+0000 7fbd44ff9700 1 -- 172.21.15.89:0/4280976113 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fbd3800cd40 con 0x7fbd480ff7d0 2023-08-25T02:31:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.664+0000 7fbd44ff9700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 0x7fbd300a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.664+0000 7fbd44ff9700 1 -- 172.21.15.89:0/4280976113 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fbd38014070 con 0x7fbd480ff7d0 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.664+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 0x7fbd300a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 0x7fbd300a2580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00b2e0 tx=0x7fbd3c005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 Processor -- start 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 -- start start 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x56451872d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x56451872ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564518695750 con 0x564518732f40 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564518686720 con 0x564518733860 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564518684040 con 0x56451872d990 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49228/0 (socket says 172.21.15.89:49228) 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 -- 172.21.15.89:0/2654973387 learned_addr learned my addr 172.21.15.89:0/2654973387 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd47fff700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x56451872ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x56451872d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 -- 172.21.15.89:0/2654973387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 msgr2=0x56451872ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x56451872ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 -- 172.21.15.89:0/2654973387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 msgr2=0x56451872d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x56451872d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 -- 172.21.15.89:0/2654973387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645184ff5d0 con 0x564518732f40 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x56451872d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:31:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.665+0000 7fbd477fe700 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbd38095420 tx=0x7fbd380996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e0e3d0af963a06 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd2dffb700 1 -- 172.21.15.89:0/2654973387 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3809cd50 con 0x564518732f40 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd2dffb700 1 -- 172.21.15.89:0/2654973387 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd380a0d30 con 0x564518732f40 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd2dffb700 1 -- 172.21.15.89:0/2654973387 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a08a0 con 0x564518732f40 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 -- 172.21.15.89:0/2654973387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 msgr2=0x564518733320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbd38095420 tx=0x7fbd380996a0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 -- 172.21.15.89:0/2654973387 shutdown_connections 2023-08-25T02:31:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x56451872ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518733320 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/2654973387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x56451872d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 -- 172.21.15.89:0/2654973387 >> 172.21.15.89:0/2654973387 conn(0x5645186a2080 msgr2=0x564518727d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 -- 172.21.15.89:0/2654973387 shutdown_connections 2023-08-25T02:31:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.666+0000 7fbd59db0b80 1 -- 172.21.15.89:0/2654973387 wait complete. 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 -- start start 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 Processor -- start 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x564518736990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x564518737410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451867f930 con 0x564518732f40 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451861db50 con 0x564518733860 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451861d850 con 0x56451872d990 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd59db0b80 10 client.0 _ll_register_callbacks cb 0x564518567a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd46ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd46ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49236/0 (socket says 172.21.15.89:49236) 2023-08-25T02:31:13.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd46ffd700 1 -- 172.21.15.89:0/1986205768 learned_addr learned my addr 172.21.15.89:0/1986205768 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x564518736990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.667+0000 7fbd47fff700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x564518737410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 -- 172.21.15.89:0/1986205768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 msgr2=0x564518736990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x564518736990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 -- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 msgr2=0x564518737410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x564518737410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 -- 172.21.15.89:0/1986205768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd481a2830 con 0x564518732f40 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd47fff700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x564518737410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00f7b0 tx=0x7fbd3c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c00d3e2e4c93ac server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c005250 con 0x564518732f40 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fbd3c0053b0 con 0x564518732f40 2023-08-25T02:31:13.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5645186fd160 con 0x564518732f40 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd3c005570 con 0x564518732f40 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 10 client.15699 fetch_fsmap learned FSMap version 37 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5645184abd80 con 0x564518732f40 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5645186fd160 con 0x564518732f40 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 37) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbd3c024750 con 0x564518732f40 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 10 client.15699 fetch_fsmap finished waiting for FSMap version 37 2023-08-25T02:31:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 20 client.15699 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.668+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187493b0 0x56451874b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.669+0000 7fbd59db0b80 4 client.15699 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.669+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564518625b40 con 0x5645187493b0 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.669+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187493b0 0x56451874b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.669+0000 7fbd26ffd700 10 client.15699 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.669+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187493b0 0x56451874b7d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83c6d29cd5414973 server_cookie=1ad53c95dfa4a289 in_seq=0 out_seq=0 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1986205768 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564518625b40 con 0x5645187493b0 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd26ffd700 10 client.15699 handle_command_reply: tid=0 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 client.15699 shutdown 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 msgr2=0x564518736ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00f7b0 tx=0x7fbd3c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 shutdown_connections 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187493b0 0x56451874b7d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518733860 0x564518737410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd26ffd700 0 client.15699 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56451872d990 0x564518736990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1986205768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518732f40 0x564518736ed0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 >> 172.21.15.89:0/1986205768 conn(0x5645186a2080 msgr2=0x564518722820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 shutdown_connections 2023-08-25T02:31:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1986205768 wait complete. 2023-08-25T02:31:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.670+0000 7fbd59db0b80 20 client.15699 trim_cache size 0 max 16384 2023-08-25T02:31:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 Processor -- start 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 -- start start 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518732f40 0x564518740340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 0x564518730a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645185018c0 con 0x56451872d990 2023-08-25T02:31:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564518734ef0 con 0x564518732f40 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564518731600 con 0x564518733860 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49242/0 (socket says 172.21.15.89:49242) 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 -- 172.21.15.89:0/3167163696 learned_addr learned my addr 172.21.15.89:0/3167163696 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd47fff700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 0x564518730a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518732f40 0x564518740340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 -- 172.21.15.89:0/3167163696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 msgr2=0x564518730a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 0x564518730a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 -- 172.21.15.89:0/3167163696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518732f40 msgr2=0x564518740340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518732f40 0x564518740340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 -- 172.21.15.89:0/3167163696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564518729f70 con 0x56451872d990 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd47fff700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 0x564518730a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd477fe700 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd38098c20 tx=0x7fbd38099790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf41fc3c8d28f202 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.685+0000 7fbd2e7fc700 1 -- 172.21.15.89:0/3167163696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3809cdd0 con 0x56451872d990 2023-08-25T02:31:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd2e7fc700 1 -- 172.21.15.89:0/3167163696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd38099180 con 0x56451872d990 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd2e7fc700 1 -- 172.21.15.89:0/3167163696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a0ae0 con 0x56451872d990 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- 172.21.15.89:0/3167163696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 msgr2=0x56451872d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd38098c20 tx=0x7fbd38099790 comp rx=0 tx=0).stop 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- 172.21.15.89:0/3167163696 shutdown_connections 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518733860 0x564518730a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56451872d990 0x56451872d450 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/3167163696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564518732f40 0x564518740340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- 172.21.15.89:0/3167163696 >> 172.21.15.89:0/3167163696 conn(0x5645186a2080 msgr2=0x564518737150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- 172.21.15.89:0/3167163696 shutdown_connections 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- 172.21.15.89:0/3167163696 wait complete. 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.686+0000 7fbd59db0b80 1 -- start start 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 Processor -- start 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56451872d990 0x56451873c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518732f40 0x56451873c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451879e590 con 0x564518733860 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451879e6d0 con 0x56451872d990 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56451879e810 con 0x564518732f40 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd59db0b80 10 client.0 _ll_register_callbacks cb 0x7fbd3c005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49252/0 (socket says 172.21.15.89:49252) 2023-08-25T02:31:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 -- 172.21.15.89:0/1870044050 learned_addr learned my addr 172.21.15.89:0/1870044050 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd46ffd700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518732f40 0x56451873c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56451872d990 0x56451873c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 -- 172.21.15.89:0/1870044050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518732f40 msgr2=0x56451873c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518732f40 0x56451873c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 -- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56451872d990 msgr2=0x56451873c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56451872d990 0x56451873c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 -- 172.21.15.89:0/1870044050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645184ff5d0 con 0x564518733860 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd47fff700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbd40007b50 tx=0x7fbd40007b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39236b3ea41efb1f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.687+0000 7fbd26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40009a70 con 0x564518733860 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fbd40017830 con 0x564518733860 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564518720600 con 0x564518733860 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd400179f0 con 0x564518733860 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 10 client.15705 fetch_fsmap learned FSMap version 37 2023-08-25T02:31:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x564518494c00 con 0x564518733860 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd4002d080 con 0x564518733860 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 37) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbd40027030 con 0x564518733860 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 10 client.15705 fetch_fsmap finished waiting for FSMap version 37 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 20 client.15705 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187a0050 0x5645187a2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 4 client.15705 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:31:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.688+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564518678910 con 0x5645187a0050 2023-08-25T02:31:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.689+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187a0050 0x5645187a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.689+0000 7fbd26ffd700 10 client.15705 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.689+0000 7fbd477fe700 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187a0050 0x5645187a2470 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37656ee4eb11f951 server_cookie=9751681765ce840d in_seq=0 out_seq=0 2023-08-25T02:31:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd26ffd700 1 -- 172.21.15.89:0/1870044050 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564518678910 con 0x5645187a0050 2023-08-25T02:31:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd26ffd700 10 client.15705 handle_command_reply: tid=0 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 client.15705 shutdown 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 msgr2=0x56451873cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbd40007b50 tx=0x7fbd40007b80 comp rx=0 tx=0).stop 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 shutdown_connections 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x5645187a0050 0x5645187a2470 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56451872d990 0x56451873c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd26ffd700 0 client.15705 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564518733860 0x56451873cda0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 --2- 172.21.15.89:0/1870044050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564518732f40 0x56451873c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 >> 172.21.15.89:0/1870044050 conn(0x5645186a2080 msgr2=0x564518730a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 shutdown_connections 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 1 -- 172.21.15.89:0/1870044050 wait complete. 2023-08-25T02:31:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.698+0000 7fbd59db0b80 20 client.15705 trim_cache size 0 max 16384 2023-08-25T02:31:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:31:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:31:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:31:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 msgr2=0x7fbd300a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 0x7fbd300a2580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00b2e0 tx=0x7fbd3c005fb0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 msgr2=0x7fbd4819b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800e9a0 tx=0x7fbd3800ecb0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 shutdown_connections 2023-08-25T02:31:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fbd300a00c0 0x7fbd300a2580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd481057e0 0x7fbd481956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd480ff7d0 0x7fbd4819b840 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 --2- 172.21.15.89:0/4280976113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd48100190 0x7fbd4819bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 >> 172.21.15.89:0/4280976113 conn(0x7fbd480fb260 msgr2=0x7fbd481032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 shutdown_connections 2023-08-25T02:31:13.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:13.699+0000 7fbd4dc0c700 1 -- 172.21.15.89:0/4280976113 wait complete. 2023-08-25T02:31:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:13 smithi089 ceph-mon[98732]: pgmap v144: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 23 op/s 2023-08-25T02:31:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:13 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:13.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:13 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:13 smithi121 ceph-mon[100099]: pgmap v144: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 23 op/s 2023-08-25T02:31:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:13 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:13 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:13.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:13 smithi122 ceph-mon[100457]: pgmap v144: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 37 KiB/s wr, 23 op/s 2023-08-25T02:31:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:13 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:13 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:14.472 INFO:teuthology.orchestra.run.smithi089.stderr:43600000 of 50000000 tuples (87%) done (elapsed 273.77 s, remaining 40.19 s) 2023-08-25T02:31:14.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: 59 slow requests, 5 included below; oldest blocked for > 188.561696 secs 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: slow request 188.458309 seconds old, received at 2023-08-25T02:28:05.933692+0000: peer_request:mds.1:1557 currently dispatched 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: slow request 188.450585 seconds old, received at 2023-08-25T02:28:05.941416+0000: peer_request:mds.1:1589 currently dispatched 2023-08-25T02:31:14.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: slow request 188.442815 seconds old, received at 2023-08-25T02:28:05.949186+0000: peer_request:mds.1:1621 currently dispatched 2023-08-25T02:31:14.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: slow request 188.434962 seconds old, received at 2023-08-25T02:28:05.957039+0000: peer_request:mds.1:1653 currently dispatched 2023-08-25T02:31:14.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:14 smithi089 ceph-mon[98732]: slow request 188.427165 seconds old, received at 2023-08-25T02:28:05.964837+0000: peer_request:mds.1:1685 currently dispatched 2023-08-25T02:31:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: 59 slow requests, 5 included below; oldest blocked for > 188.561696 secs 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: slow request 188.458309 seconds old, received at 2023-08-25T02:28:05.933692+0000: peer_request:mds.1:1557 currently dispatched 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: slow request 188.450585 seconds old, received at 2023-08-25T02:28:05.941416+0000: peer_request:mds.1:1589 currently dispatched 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: slow request 188.442815 seconds old, received at 2023-08-25T02:28:05.949186+0000: peer_request:mds.1:1621 currently dispatched 2023-08-25T02:31:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: slow request 188.434962 seconds old, received at 2023-08-25T02:28:05.957039+0000: peer_request:mds.1:1653 currently dispatched 2023-08-25T02:31:14.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:14 smithi121 ceph-mon[100099]: slow request 188.427165 seconds old, received at 2023-08-25T02:28:05.964837+0000: peer_request:mds.1:1685 currently dispatched 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: 59 slow requests, 5 included below; oldest blocked for > 188.561696 secs 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: slow request 188.458309 seconds old, received at 2023-08-25T02:28:05.933692+0000: peer_request:mds.1:1557 currently dispatched 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: slow request 188.450585 seconds old, received at 2023-08-25T02:28:05.941416+0000: peer_request:mds.1:1589 currently dispatched 2023-08-25T02:31:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: slow request 188.442815 seconds old, received at 2023-08-25T02:28:05.949186+0000: peer_request:mds.1:1621 currently dispatched 2023-08-25T02:31:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: slow request 188.434962 seconds old, received at 2023-08-25T02:28:05.957039+0000: peer_request:mds.1:1653 currently dispatched 2023-08-25T02:31:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:14 smithi122 ceph-mon[100457]: slow request 188.427165 seconds old, received at 2023-08-25T02:28:05.964837+0000: peer_request:mds.1:1685 currently dispatched 2023-08-25T02:31:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: 535 slow requests, 5 included below; oldest blocked for > 198.364004 secs 2023-08-25T02:31:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: slow request 197.609043 seconds old, received at 2023-08-25T02:27:56.375819+0000: client_request(mds.1:1867 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: slow request 197.578223 seconds old, received at 2023-08-25T02:27:56.406639+0000: client_request(mds.1:1899 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: slow request 197.546884 seconds old, received at 2023-08-25T02:27:56.437977+0000: client_request(mds.1:1931 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: slow request 197.515429 seconds old, received at 2023-08-25T02:27:56.469433+0000: client_request(mds.1:1963 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: slow request 197.484617 seconds old, received at 2023-08-25T02:27:56.500245+0000: client_request(mds.1:1995 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:15 smithi089 ceph-mon[98732]: pgmap v145: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 17 op/s 2023-08-25T02:31:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: 535 slow requests, 5 included below; oldest blocked for > 198.364004 secs 2023-08-25T02:31:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: slow request 197.609043 seconds old, received at 2023-08-25T02:27:56.375819+0000: client_request(mds.1:1867 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: slow request 197.578223 seconds old, received at 2023-08-25T02:27:56.406639+0000: client_request(mds.1:1899 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: slow request 197.546884 seconds old, received at 2023-08-25T02:27:56.437977+0000: client_request(mds.1:1931 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: slow request 197.515429 seconds old, received at 2023-08-25T02:27:56.469433+0000: client_request(mds.1:1963 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: slow request 197.484617 seconds old, received at 2023-08-25T02:27:56.500245+0000: client_request(mds.1:1995 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:15 smithi121 ceph-mon[100099]: pgmap v145: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 17 op/s 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: 535 slow requests, 5 included below; oldest blocked for > 198.364004 secs 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: slow request 197.609043 seconds old, received at 2023-08-25T02:27:56.375819+0000: client_request(mds.1:1867 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: slow request 197.578223 seconds old, received at 2023-08-25T02:27:56.406639+0000: client_request(mds.1:1899 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: slow request 197.546884 seconds old, received at 2023-08-25T02:27:56.437977+0000: client_request(mds.1:1931 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: slow request 197.515429 seconds old, received at 2023-08-25T02:27:56.469433+0000: client_request(mds.1:1963 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: slow request 197.484617 seconds old, received at 2023-08-25T02:27:56.500245+0000: client_request(mds.1:1995 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:15.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:15 smithi122 ceph-mon[100457]: pgmap v145: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 KiB/s wr, 17 op/s 2023-08-25T02:31:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:17 smithi089 ceph-mon[98732]: pgmap v146: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2023-08-25T02:31:17.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:17 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:17.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:17 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:17 smithi121 ceph-mon[100099]: pgmap v146: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2023-08-25T02:31:17.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:17 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:17.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:17 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:17.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:17 smithi122 ceph-mon[100457]: pgmap v146: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 25 op/s 2023-08-25T02:31:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:17 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:17 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:19.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: pgmap v147: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:19.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: 1362 slow requests, 5 included below; oldest blocked for > 193.561779 secs 2023-08-25T02:31:19.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: slow request 193.419476 seconds old, received at 2023-08-25T02:28:05.972608+0000: peer_request:mds.1:1717 currently dispatched 2023-08-25T02:31:19.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: slow request 193.411804 seconds old, received at 2023-08-25T02:28:05.980280+0000: peer_request:mds.1:1749 currently dispatched 2023-08-25T02:31:19.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: slow request 193.404643 seconds old, received at 2023-08-25T02:28:05.987441+0000: peer_request:mds.1:1781 currently dispatched 2023-08-25T02:31:19.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: slow request 193.397460 seconds old, received at 2023-08-25T02:28:05.994625+0000: peer_request:mds.1:1813 currently dispatched 2023-08-25T02:31:19.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:19 smithi089 ceph-mon[98732]: slow request 193.389907 seconds old, received at 2023-08-25T02:28:06.002177+0000: peer_request:mds.1:1845 currently dispatched 2023-08-25T02:31:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: pgmap v147: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: 1362 slow requests, 5 included below; oldest blocked for > 193.561779 secs 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: slow request 193.419476 seconds old, received at 2023-08-25T02:28:05.972608+0000: peer_request:mds.1:1717 currently dispatched 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: slow request 193.411804 seconds old, received at 2023-08-25T02:28:05.980280+0000: peer_request:mds.1:1749 currently dispatched 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: slow request 193.404643 seconds old, received at 2023-08-25T02:28:05.987441+0000: peer_request:mds.1:1781 currently dispatched 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: slow request 193.397460 seconds old, received at 2023-08-25T02:28:05.994625+0000: peer_request:mds.1:1813 currently dispatched 2023-08-25T02:31:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:19 smithi121 ceph-mon[100099]: slow request 193.389907 seconds old, received at 2023-08-25T02:28:06.002177+0000: peer_request:mds.1:1845 currently dispatched 2023-08-25T02:31:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: pgmap v147: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: 1362 slow requests, 5 included below; oldest blocked for > 193.561779 secs 2023-08-25T02:31:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: slow request 193.419476 seconds old, received at 2023-08-25T02:28:05.972608+0000: peer_request:mds.1:1717 currently dispatched 2023-08-25T02:31:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: slow request 193.411804 seconds old, received at 2023-08-25T02:28:05.980280+0000: peer_request:mds.1:1749 currently dispatched 2023-08-25T02:31:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: slow request 193.404643 seconds old, received at 2023-08-25T02:28:05.987441+0000: peer_request:mds.1:1781 currently dispatched 2023-08-25T02:31:19.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: slow request 193.397460 seconds old, received at 2023-08-25T02:28:05.994625+0000: peer_request:mds.1:1813 currently dispatched 2023-08-25T02:31:19.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:19 smithi122 ceph-mon[100457]: slow request 193.389907 seconds old, received at 2023-08-25T02:28:06.002177+0000: peer_request:mds.1:1845 currently dispatched 2023-08-25T02:31:20.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: 177 slow requests, 5 included below; oldest blocked for > 203.391138 secs 2023-08-25T02:31:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: slow request 202.482173 seconds old, received at 2023-08-25T02:27:56.529822+0000: client_request(mds.1:2027 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: slow request 202.452785 seconds old, received at 2023-08-25T02:27:56.559210+0000: client_request(mds.1:2059 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: slow request 202.423391 seconds old, received at 2023-08-25T02:27:56.588604+0000: client_request(mds.1:2091 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: slow request 202.393632 seconds old, received at 2023-08-25T02:27:56.618363+0000: client_request(mds.1:2123 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:20 smithi089 ceph-mon[98732]: slow request 202.364169 seconds old, received at 2023-08-25T02:27:56.647826+0000: client_request(mds.1:2155 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: 177 slow requests, 5 included below; oldest blocked for > 203.391138 secs 2023-08-25T02:31:20.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: slow request 202.482173 seconds old, received at 2023-08-25T02:27:56.529822+0000: client_request(mds.1:2027 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: slow request 202.452785 seconds old, received at 2023-08-25T02:27:56.559210+0000: client_request(mds.1:2059 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: slow request 202.423391 seconds old, received at 2023-08-25T02:27:56.588604+0000: client_request(mds.1:2091 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: slow request 202.393632 seconds old, received at 2023-08-25T02:27:56.618363+0000: client_request(mds.1:2123 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:20 smithi121 ceph-mon[100099]: slow request 202.364169 seconds old, received at 2023-08-25T02:27:56.647826+0000: client_request(mds.1:2155 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: 177 slow requests, 5 included below; oldest blocked for > 203.391138 secs 2023-08-25T02:31:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: slow request 202.482173 seconds old, received at 2023-08-25T02:27:56.529822+0000: client_request(mds.1:2027 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: slow request 202.452785 seconds old, received at 2023-08-25T02:27:56.559210+0000: client_request(mds.1:2059 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: slow request 202.423391 seconds old, received at 2023-08-25T02:27:56.588604+0000: client_request(mds.1:2091 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: slow request 202.393632 seconds old, received at 2023-08-25T02:27:56.618363+0000: client_request(mds.1:2123 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:20.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:20 smithi122 ceph-mon[100457]: slow request 202.364169 seconds old, received at 2023-08-25T02:27:56.647826+0000: client_request(mds.1:2155 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:43700000 of 50000000 tuples (87%) done (elapsed 280.46 s, remaining 40.43 s) 2023-08-25T02:31:21.240 INFO:teuthology.orchestra.run.smithi089.stderr:43800000 of 50000000 tuples (87%) done (elapsed 280.54 s, remaining 39.71 s) 2023-08-25T02:31:21.355 INFO:teuthology.orchestra.run.smithi089.stderr:43900000 of 50000000 tuples (87%) done (elapsed 280.65 s, remaining 39.00 s) 2023-08-25T02:31:21.445 INFO:teuthology.orchestra.run.smithi089.stderr:44000000 of 50000000 tuples (88%) done (elapsed 280.74 s, remaining 38.28 s) 2023-08-25T02:31:21.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:21 smithi089 ceph-mon[98732]: pgmap v148: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:21.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:21 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:21.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:21 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:21.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:21 smithi121 ceph-mon[100099]: pgmap v148: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:21.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:21 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:21.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:21 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:21.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:21 smithi122 ceph-mon[100457]: pgmap v148: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 10 op/s 2023-08-25T02:31:21.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:21 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:21.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:21 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:22 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:22 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:22.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:22 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:22.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:22 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:22.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:22 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:22.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:22 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:23.837 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:23 smithi089 ceph-mon[98732]: pgmap v149: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 16 op/s 2023-08-25T02:31:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:23 smithi121 ceph-mon[100099]: pgmap v149: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 16 op/s 2023-08-25T02:31:23.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:23 smithi122 ceph-mon[100457]: pgmap v149: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 16 op/s 2023-08-25T02:31:24.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: 116 slow requests, 5 included below; oldest blocked for > 198.561898 secs 2023-08-25T02:31:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: slow request 198.382235 seconds old, received at 2023-08-25T02:28:06.009969+0000: peer_request:mds.1:1877 currently dispatched 2023-08-25T02:31:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: slow request 198.374504 seconds old, received at 2023-08-25T02:28:06.017699+0000: peer_request:mds.1:1909 currently dispatched 2023-08-25T02:31:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: slow request 198.366758 seconds old, received at 2023-08-25T02:28:06.025445+0000: peer_request:mds.1:1941 currently dispatched 2023-08-25T02:31:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: slow request 198.358914 seconds old, received at 2023-08-25T02:28:06.033290+0000: peer_request:mds.1:1973 currently dispatched 2023-08-25T02:31:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:24 smithi089 ceph-mon[98732]: slow request 198.350988 seconds old, received at 2023-08-25T02:28:06.041215+0000: peer_request:mds.1:2005 currently dispatched 2023-08-25T02:31:24.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: 116 slow requests, 5 included below; oldest blocked for > 198.561898 secs 2023-08-25T02:31:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: slow request 198.382235 seconds old, received at 2023-08-25T02:28:06.009969+0000: peer_request:mds.1:1877 currently dispatched 2023-08-25T02:31:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: slow request 198.374504 seconds old, received at 2023-08-25T02:28:06.017699+0000: peer_request:mds.1:1909 currently dispatched 2023-08-25T02:31:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: slow request 198.366758 seconds old, received at 2023-08-25T02:28:06.025445+0000: peer_request:mds.1:1941 currently dispatched 2023-08-25T02:31:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: slow request 198.358914 seconds old, received at 2023-08-25T02:28:06.033290+0000: peer_request:mds.1:1973 currently dispatched 2023-08-25T02:31:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:24 smithi121 ceph-mon[100099]: slow request 198.350988 seconds old, received at 2023-08-25T02:28:06.041215+0000: peer_request:mds.1:2005 currently dispatched 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: 116 slow requests, 5 included below; oldest blocked for > 198.561898 secs 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: slow request 198.382235 seconds old, received at 2023-08-25T02:28:06.009969+0000: peer_request:mds.1:1877 currently dispatched 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: slow request 198.374504 seconds old, received at 2023-08-25T02:28:06.017699+0000: peer_request:mds.1:1909 currently dispatched 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: slow request 198.366758 seconds old, received at 2023-08-25T02:28:06.025445+0000: peer_request:mds.1:1941 currently dispatched 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: slow request 198.358914 seconds old, received at 2023-08-25T02:28:06.033290+0000: peer_request:mds.1:1973 currently dispatched 2023-08-25T02:31:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:24 smithi122 ceph-mon[100457]: slow request 198.350988 seconds old, received at 2023-08-25T02:28:06.041215+0000: peer_request:mds.1:2005 currently dispatched 2023-08-25T02:31:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: 236 slow requests, 5 included below; oldest blocked for > 208.503652 secs 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: slow request 207.447215 seconds old, received at 2023-08-25T02:27:56.677294+0000: client_request(mds.1:2187 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: slow request 207.417565 seconds old, received at 2023-08-25T02:27:56.706944+0000: client_request(mds.1:2219 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: slow request 207.388252 seconds old, received at 2023-08-25T02:27:56.736257+0000: client_request(mds.1:2251 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: slow request 207.358477 seconds old, received at 2023-08-25T02:27:56.766032+0000: client_request(mds.1:2283 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: slow request 207.328680 seconds old, received at 2023-08-25T02:27:56.795829+0000: client_request(mds.1:2315 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:25 smithi121 ceph-mon[100099]: pgmap v150: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.4 MiB/s wr, 13 op/s 2023-08-25T02:31:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: 236 slow requests, 5 included below; oldest blocked for > 208.503652 secs 2023-08-25T02:31:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: slow request 207.447215 seconds old, received at 2023-08-25T02:27:56.677294+0000: client_request(mds.1:2187 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: slow request 207.417565 seconds old, received at 2023-08-25T02:27:56.706944+0000: client_request(mds.1:2219 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: slow request 207.388252 seconds old, received at 2023-08-25T02:27:56.736257+0000: client_request(mds.1:2251 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: slow request 207.358477 seconds old, received at 2023-08-25T02:27:56.766032+0000: client_request(mds.1:2283 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: slow request 207.328680 seconds old, received at 2023-08-25T02:27:56.795829+0000: client_request(mds.1:2315 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:25.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:25 smithi122 ceph-mon[100457]: pgmap v150: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.4 MiB/s wr, 13 op/s 2023-08-25T02:31:26.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: 236 slow requests, 5 included below; oldest blocked for > 208.503652 secs 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: slow request 207.447215 seconds old, received at 2023-08-25T02:27:56.677294+0000: client_request(mds.1:2187 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: slow request 207.417565 seconds old, received at 2023-08-25T02:27:56.706944+0000: client_request(mds.1:2219 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: slow request 207.388252 seconds old, received at 2023-08-25T02:27:56.736257+0000: client_request(mds.1:2251 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: slow request 207.358477 seconds old, received at 2023-08-25T02:27:56.766032+0000: client_request(mds.1:2283 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: slow request 207.328680 seconds old, received at 2023-08-25T02:27:56.795829+0000: client_request(mds.1:2315 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:26.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:25 smithi089 ceph-mon[98732]: pgmap v150: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.4 MiB/s wr, 13 op/s 2023-08-25T02:31:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- 172.21.15.89:0/2630451860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 msgr2=0x7fd5b01006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 --2- 172.21.15.89:0/2630451860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b01006e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40099c0 tx=0x7fd5a4009cd0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- 172.21.15.89:0/2630451860 shutdown_connections 2023-08-25T02:31:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 --2- 172.21.15.89:0/2630451860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0100c20 0x7fd5b010c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 --2- 172.21.15.89:0/2630451860 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0105910 0x7fd5b0105cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 --2- 172.21.15.89:0/2630451860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b01006e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- 172.21.15.89:0/2630451860 >> 172.21.15.89:0/2630451860 conn(0x7fd5b0074a60 msgr2=0x7fd5b0074e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- 172.21.15.89:0/2630451860 shutdown_connections 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- 172.21.15.89:0/2630451860 wait complete. 2023-08-25T02:31:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 Processor -- start 2023-08-25T02:31:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.817+0000 7fd5b6124700 1 -- start start 2023-08-25T02:31:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 0x7fd5b019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0105910 0x7fd5b0196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0196db0 con 0x7fd5b0100260 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0196f20 con 0x7fd5b0105910 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5b6124700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b01970e0 con 0x7fd5b0100c20 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49264/0 (socket says 172.21.15.89:49264) 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 -- 172.21.15.89:0/3449703895 learned_addr learned my addr 172.21.15.89:0/3449703895 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 0x7fd5b019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5affff700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0105910 0x7fd5b0196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 -- 172.21.15.89:0/3449703895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 msgr2=0x7fd5b019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 0x7fd5b019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 -- 172.21.15.89:0/3449703895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0105910 msgr2=0x7fd5b0196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0105910 0x7fd5b0196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 -- 172.21.15.89:0/3449703895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40095f0 con 0x7fd5b0100260 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 0x7fd5b019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.818+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000e970 tx=0x7fd5a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.819+0000 7fd5acff9700 1 -- 172.21.15.89:0/3449703895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0009790 con 0x7fd5b0100260 2023-08-25T02:31:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.819+0000 7fd5acff9700 1 -- 172.21.15.89:0/3449703895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5a000c7e0 con 0x7fd5b0100260 2023-08-25T02:31:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.819+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b01973c0 con 0x7fd5b0100260 2023-08-25T02:31:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.819+0000 7fd5acff9700 1 -- 172.21.15.89:0/3449703895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0003c70 con 0x7fd5b0100260 2023-08-25T02:31:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.819+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b01a38d0 con 0x7fd5b0100260 2023-08-25T02:31:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.821+0000 7fd5acff9700 1 -- 172.21.15.89:0/3449703895 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fd5a000cc50 con 0x7fd5b0100260 2023-08-25T02:31:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.822+0000 7fd5acff9700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 0x7fd5980a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.822+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 0x7fd5980a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.822+0000 7fd5acff9700 1 -- 172.21.15.89:0/3449703895 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fd5a0014070 con 0x7fd5b0100260 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.822+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 0x7fd5980a2580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4005fb0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 Processor -- start 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 -- start start 2023-08-25T02:31:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 0x55953aad4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aace990 0x55953aacedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aa36750 con 0x55953aad4860 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aa27720 con 0x55953aad3f40 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aa25040 con 0x55953aace990 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5affff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aace990 0x55953aacedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49274/0 (socket says 172.21.15.89:49274) 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1154830163 learned_addr learned my addr 172.21.15.89:0/1154830163 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 0x55953aad4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1154830163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aace990 msgr2=0x55953aacedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aace990 0x55953aacedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1154830163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 msgr2=0x55953aad4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 0x55953aad4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1154830163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55953a8a05d0 con 0x55953aad4860 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 0x55953aad4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.823+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40099c0 tx=0x7fd5a4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf14f0fb73612c64 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd595ffb700 1 -- 172.21.15.89:0/1154830163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a402a070 con 0x55953aad4860 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd595ffb700 1 -- 172.21.15.89:0/1154830163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5a401e040 con 0x55953aad4860 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd595ffb700 1 -- 172.21.15.89:0/1154830163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400fc00 con 0x55953aad4860 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1154830163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 msgr2=0x55953aace450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40099c0 tx=0x7fd5a4020000 comp rx=0 tx=0).stop 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1154830163 shutdown_connections 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad3f40 0x55953aad4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aace990 0x55953aacedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1154830163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad4860 0x55953aace450 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1154830163 >> 172.21.15.89:0/1154830163 conn(0x55953aa43080 msgr2=0x55953aac8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1154830163 shutdown_connections 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.824+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1154830163 wait complete. 2023-08-25T02:31:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 -- start start 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 Processor -- start 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aada580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aad7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aa20930 con 0x55953aad3f40 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953a9beb50 con 0x55953aace990 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953a9be850 con 0x55953aad4860 2023-08-25T02:31:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5c22c8b80 10 client.0 _ll_register_callbacks cb 0x55953a908a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.825+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aada580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49280/0 (socket says 172.21.15.89:49280) 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5affff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aad7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aada580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:39688/0 (socket says 172.21.15.89:39688) 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1799815757 learned_addr learned my addr 172.21.15.89:0/1799815757 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1799815757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 msgr2=0x55953aad7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aad7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 msgr2=0x55953aada580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aada580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 -- 172.21.15.89:0/1799815757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b010cf90 con 0x55953aad3f40 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5affff700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aad7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f69aae4a7010442 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd58effd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a402a070 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fd5a4005460 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55953aa9e160 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a4005620 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5c22c8b80 10 client.15717 fetch_fsmap learned FSMap version 41 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55953a84cd80 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.826+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55953aa9e160 con 0x55953aad3f40 2023-08-25T02:31:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5a4024410 con 0x55953aad3f40 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 10 client.15717 fetch_fsmap finished waiting for FSMap version 41 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 20 client.15717 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953aae7c20 0x55953aaea010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 4 client.15717 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55953a9c6b40 con 0x55953aae7c20 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953aae7c20 0x55953aaea010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953aae7c20 0x55953aaea010 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=993e298d9b3e15c8 server_cookie=f39693a9771a377e in_seq=0 out_seq=0 2023-08-25T02:31:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.827+0000 7fd58effd700 10 client.15717 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd58effd700 1 -- 172.21.15.89:0/1799815757 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55953a9c6b40 con 0x55953aae7c20 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd58effd700 10 client.15717 handle_command_reply: tid=0 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 client.15717 shutdown 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 msgr2=0x55953aad7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009310 comp rx=0 tx=0).stop 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 shutdown_connections 2023-08-25T02:31:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953aae7c20 0x55953aaea010 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aada580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aad7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd58effd700 0 client.15717 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/1799815757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aad7800 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 >> 172.21.15.89:0/1799815757 conn(0x55953aa43080 msgr2=0x55953aac3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 shutdown_connections 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/1799815757 wait complete. 2023-08-25T02:31:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.828+0000 7fd5c22c8b80 20 client.15717 trim_cache size 0 max 16384 2023-08-25T02:31:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 Processor -- start 2023-08-25T02:31:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 -- start start 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aace450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aada120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953a8a28c0 con 0x55953aad3f40 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aaca740 con 0x55953aace990 2023-08-25T02:31:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aac9250 con 0x55953aad4860 2023-08-25T02:31:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5aeffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49288/0 (socket says 172.21.15.89:49288) 2023-08-25T02:31:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5aeffd700 1 -- 172.21.15.89:0/3539104788 learned_addr learned my addr 172.21.15.89:0/3539104788 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5affff700 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aada120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 -- 172.21.15.89:0/3539104788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 msgr2=0x55953aada120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.843+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aace450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aada120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 -- 172.21.15.89:0/3539104788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 msgr2=0x55953aace450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aace450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 -- 172.21.15.89:0/3539104788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55953a835c00 con 0x55953aad3f40 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40247f0 tx=0x7fd5a4005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0c7d7b15d59752 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5967fc700 1 -- 172.21.15.89:0/3539104788 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a402a070 con 0x55953aad3f40 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5967fc700 1 -- 172.21.15.89:0/3539104788 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5a401e040 con 0x55953aad3f40 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5967fc700 1 -- 172.21.15.89:0/3539104788 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4024bd0 con 0x55953aad3f40 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/3539104788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 msgr2=0x55953aadf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40247f0 tx=0x7fd5a4005940 comp rx=0 tx=0).stop 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/3539104788 shutdown_connections 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aace990 0x55953aace450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad4860 0x55953aada120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/3539104788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aad3f40 0x55953aadf210 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.844+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/3539104788 >> 172.21.15.89:0/3539104788 conn(0x55953aa43080 msgr2=0x55953aad19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/3539104788 shutdown_connections 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/3539104788 wait complete. 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 -- start start 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 Processor -- start 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad3f40 0x55953aacf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.845+0000 7fd5c22c8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad4860 0x55953aacfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aad0510 con 0x55953aace990 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aad0680 con 0x55953aad4860 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55953aad0840 con 0x55953aad3f40 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 10 client.0 _ll_register_callbacks cb 0x7fd5a8003a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49292/0 (socket says 172.21.15.89:49292) 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 -- 172.21.15.89:0/2758753989 learned_addr learned my addr 172.21.15.89:0/2758753989 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad3f40 0x55953aacf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5affff700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad4860 0x55953aacfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 -- 172.21.15.89:0/2758753989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad3f40 msgr2=0x55953aacf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad3f40 0x55953aacf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 -- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad4860 msgr2=0x55953aacfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad4860 0x55953aacfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 -- 172.21.15.89:0/2758753989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55953a84cd80 con 0x55953aace990 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5af7fe700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0095410 tx=0x7fd5a00b8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eced2e93db68f7f2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd58effd700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0098f10 con 0x55953aace990 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7fd5a009cd40 con 0x55953aace990 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55953aac1600 con 0x55953aace990 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a00a0d20 con 0x55953aace990 2023-08-25T02:31:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 10 client.15723 fetch_fsmap learned FSMap version 41 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55953a8731f0 con 0x55953aace990 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.846+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55953aac1600 con 0x55953aace990 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5a00b2070 con 0x55953aace990 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 10 client.15723 fetch_fsmap finished waiting for FSMap version 41 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 20 client.15723 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953ab3c2d0 0x55953ab3e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 4 client.15723 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:31:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55953aa19910 con 0x55953ab3c2d0 2023-08-25T02:31:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953ab3c2d0 0x55953ab3e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd5aeffd700 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953ab3c2d0 0x55953ab3e6f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8818afc7f941c86 server_cookie=d0043c4b4c177630 in_seq=0 out_seq=0 2023-08-25T02:31:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.847+0000 7fd58effd700 10 client.15723 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.893+0000 7fd58effd700 1 -- 172.21.15.89:0/2758753989 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55953aa19910 con 0x55953ab3c2d0 2023-08-25T02:31:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.893+0000 7fd58effd700 10 client.15723 handle_command_reply: tid=0 2023-08-25T02:31:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.893+0000 7fd5c22c8b80 1 client.15723 shutdown 2023-08-25T02:31:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 msgr2=0x55953aad7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0095410 tx=0x7fd5a00b8040 comp rx=0 tx=0).stop 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 shutdown_connections 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x55953ab3c2d0 0x55953ab3e6f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55953aad4860 0x55953aacfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55953aace990 0x55953aad7760 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd58effd700 0 client.15723 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 --2- 172.21.15.89:0/2758753989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55953aad3f40 0x55953aacf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 >> 172.21.15.89:0/2758753989 conn(0x55953aa43080 msgr2=0x55953aad4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 shutdown_connections 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 1 -- 172.21.15.89:0/2758753989 wait complete. 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5c22c8b80 20 client.15723 trim_cache size 0 max 16384 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:31:26.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:31:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:31:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 msgr2=0x7fd5980a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 0x7fd5980a2580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4005fb0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 msgr2=0x7fd5b019c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000e970 tx=0x7fd5a000ec80 comp rx=0 tx=0).stop 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 shutdown_connections 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd5980a00c0 0x7fd5980a2580 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5b0105910 0x7fd5b0196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5b0100260 0x7fd5b019c8e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 --2- 172.21.15.89:0/3449703895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5b0100c20 0x7fd5b019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.894+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 >> 172.21.15.89:0/3449703895 conn(0x7fd5b0074a60 msgr2=0x7fd5b0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.895+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 shutdown_connections 2023-08-25T02:31:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:26.895+0000 7fd5b6124700 1 -- 172.21.15.89:0/3449703895 wait complete. 2023-08-25T02:31:26.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:26 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:26.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:26 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:26.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:26 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:26.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:26 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:27.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:26 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:27.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:26 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:27 smithi121 ceph-mon[100099]: pgmap v151: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:31:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:27 smithi122 ceph-mon[100457]: pgmap v151: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:31:28.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:27 smithi089 ceph-mon[98732]: pgmap v151: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.7 MiB/s wr, 18 op/s 2023-08-25T02:31:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: pgmap v152: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: 351 slow requests, 5 included below; oldest blocked for > 203.562007 secs 2023-08-25T02:31:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: slow request 203.343822 seconds old, received at 2023-08-25T02:28:06.048491+0000: peer_request:mds.1:2037 currently dispatched 2023-08-25T02:31:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: slow request 203.336975 seconds old, received at 2023-08-25T02:28:06.055338+0000: peer_request:mds.1:2069 currently dispatched 2023-08-25T02:31:29.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: slow request 203.330174 seconds old, received at 2023-08-25T02:28:06.062138+0000: peer_request:mds.1:2101 currently dispatched 2023-08-25T02:31:29.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: slow request 203.322911 seconds old, received at 2023-08-25T02:28:06.069401+0000: peer_request:mds.1:2133 currently dispatched 2023-08-25T02:31:29.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:29 smithi121 ceph-mon[100099]: slow request 203.315658 seconds old, received at 2023-08-25T02:28:06.076655+0000: peer_request:mds.1:2165 currently dispatched 2023-08-25T02:31:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: pgmap v152: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:29.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: 351 slow requests, 5 included below; oldest blocked for > 203.562007 secs 2023-08-25T02:31:29.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: slow request 203.343822 seconds old, received at 2023-08-25T02:28:06.048491+0000: peer_request:mds.1:2037 currently dispatched 2023-08-25T02:31:29.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: slow request 203.336975 seconds old, received at 2023-08-25T02:28:06.055338+0000: peer_request:mds.1:2069 currently dispatched 2023-08-25T02:31:29.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: slow request 203.330174 seconds old, received at 2023-08-25T02:28:06.062138+0000: peer_request:mds.1:2101 currently dispatched 2023-08-25T02:31:29.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: slow request 203.322911 seconds old, received at 2023-08-25T02:28:06.069401+0000: peer_request:mds.1:2133 currently dispatched 2023-08-25T02:31:29.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:29 smithi122 ceph-mon[100457]: slow request 203.315658 seconds old, received at 2023-08-25T02:28:06.076655+0000: peer_request:mds.1:2165 currently dispatched 2023-08-25T02:31:30.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: pgmap v152: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:30.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:30.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:30.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: 351 slow requests, 5 included below; oldest blocked for > 203.562007 secs 2023-08-25T02:31:30.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: slow request 203.343822 seconds old, received at 2023-08-25T02:28:06.048491+0000: peer_request:mds.1:2037 currently dispatched 2023-08-25T02:31:30.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: slow request 203.336975 seconds old, received at 2023-08-25T02:28:06.055338+0000: peer_request:mds.1:2069 currently dispatched 2023-08-25T02:31:30.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: slow request 203.330174 seconds old, received at 2023-08-25T02:28:06.062138+0000: peer_request:mds.1:2101 currently dispatched 2023-08-25T02:31:30.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: slow request 203.322911 seconds old, received at 2023-08-25T02:28:06.069401+0000: peer_request:mds.1:2133 currently dispatched 2023-08-25T02:31:30.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:29 smithi089 ceph-mon[98732]: slow request 203.315658 seconds old, received at 2023-08-25T02:28:06.076655+0000: peer_request:mds.1:2165 currently dispatched 2023-08-25T02:31:30.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: 468 slow requests, 5 included below; oldest blocked for > 213.672316 secs 2023-08-25T02:31:30.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: slow request 212.467948 seconds old, received at 2023-08-25T02:27:56.825225+0000: client_request(mds.1:2347 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: slow request 212.437060 seconds old, received at 2023-08-25T02:27:56.856114+0000: client_request(mds.1:2379 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: slow request 212.405247 seconds old, received at 2023-08-25T02:27:56.887927+0000: client_request(mds.1:2411 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: slow request 212.373438 seconds old, received at 2023-08-25T02:27:56.919736+0000: client_request(mds.1:2443 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:30 smithi121 ceph-mon[100099]: slow request 212.341479 seconds old, received at 2023-08-25T02:27:56.951694+0000: client_request(mds.1:2475 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: 468 slow requests, 5 included below; oldest blocked for > 213.672316 secs 2023-08-25T02:31:30.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: slow request 212.467948 seconds old, received at 2023-08-25T02:27:56.825225+0000: client_request(mds.1:2347 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: slow request 212.437060 seconds old, received at 2023-08-25T02:27:56.856114+0000: client_request(mds.1:2379 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: slow request 212.405247 seconds old, received at 2023-08-25T02:27:56.887927+0000: client_request(mds.1:2411 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: slow request 212.373438 seconds old, received at 2023-08-25T02:27:56.919736+0000: client_request(mds.1:2443 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:30.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:30 smithi122 ceph-mon[100457]: slow request 212.341479 seconds old, received at 2023-08-25T02:27:56.951694+0000: client_request(mds.1:2475 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: 468 slow requests, 5 included below; oldest blocked for > 213.672316 secs 2023-08-25T02:31:31.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: slow request 212.467948 seconds old, received at 2023-08-25T02:27:56.825225+0000: client_request(mds.1:2347 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: slow request 212.437060 seconds old, received at 2023-08-25T02:27:56.856114+0000: client_request(mds.1:2379 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: slow request 212.405247 seconds old, received at 2023-08-25T02:27:56.887927+0000: client_request(mds.1:2411 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: slow request 212.373438 seconds old, received at 2023-08-25T02:27:56.919736+0000: client_request(mds.1:2443 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:30 smithi089 ceph-mon[98732]: slow request 212.341479 seconds old, received at 2023-08-25T02:27:56.951694+0000: client_request(mds.1:2475 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:44100000 of 50000000 tuples (88%) done (elapsed 290.54 s, remaining 38.87 s) 2023-08-25T02:31:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:44200000 of 50000000 tuples (88%) done (elapsed 290.94 s, remaining 38.18 s) 2023-08-25T02:31:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:44300000 of 50000000 tuples (88%) done (elapsed 291.01 s, remaining 37.44 s) 2023-08-25T02:31:31.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:31 smithi121 ceph-mon[100099]: pgmap v153: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:31.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:31 smithi122 ceph-mon[100457]: pgmap v153: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:32.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:31 smithi089 ceph-mon[98732]: pgmap v153: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.0 MiB/s wr, 10 op/s 2023-08-25T02:31:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:33 smithi121 ceph-mon[100099]: pgmap v154: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 16 op/s 2023-08-25T02:31:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:33 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:33 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:33 smithi122 ceph-mon[100457]: pgmap v154: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 16 op/s 2023-08-25T02:31:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:33 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:33 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:34.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:33 smithi089 ceph-mon[98732]: pgmap v154: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.4 MiB/s wr, 16 op/s 2023-08-25T02:31:34.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:33 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:34.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:33 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:34.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: 700 slow requests, 5 included below; oldest blocked for > 208.562108 secs 2023-08-25T02:31:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: slow request 208.308532 seconds old, received at 2023-08-25T02:28:06.083881+0000: peer_request:mds.1:2197 currently dispatched 2023-08-25T02:31:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: slow request 208.301405 seconds old, received at 2023-08-25T02:28:06.091009+0000: peer_request:mds.1:2229 currently dispatched 2023-08-25T02:31:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: slow request 208.294202 seconds old, received at 2023-08-25T02:28:06.098212+0000: peer_request:mds.1:2261 currently dispatched 2023-08-25T02:31:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: slow request 208.286964 seconds old, received at 2023-08-25T02:28:06.105449+0000: peer_request:mds.1:2293 currently dispatched 2023-08-25T02:31:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:34 smithi121 ceph-mon[100099]: slow request 208.279733 seconds old, received at 2023-08-25T02:28:06.112681+0000: peer_request:mds.1:2325 currently dispatched 2023-08-25T02:31:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: 700 slow requests, 5 included below; oldest blocked for > 208.562108 secs 2023-08-25T02:31:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: slow request 208.308532 seconds old, received at 2023-08-25T02:28:06.083881+0000: peer_request:mds.1:2197 currently dispatched 2023-08-25T02:31:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: slow request 208.301405 seconds old, received at 2023-08-25T02:28:06.091009+0000: peer_request:mds.1:2229 currently dispatched 2023-08-25T02:31:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: slow request 208.294202 seconds old, received at 2023-08-25T02:28:06.098212+0000: peer_request:mds.1:2261 currently dispatched 2023-08-25T02:31:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: slow request 208.286964 seconds old, received at 2023-08-25T02:28:06.105449+0000: peer_request:mds.1:2293 currently dispatched 2023-08-25T02:31:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:34 smithi122 ceph-mon[100457]: slow request 208.279733 seconds old, received at 2023-08-25T02:28:06.112681+0000: peer_request:mds.1:2325 currently dispatched 2023-08-25T02:31:35.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: 700 slow requests, 5 included below; oldest blocked for > 208.562108 secs 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: slow request 208.308532 seconds old, received at 2023-08-25T02:28:06.083881+0000: peer_request:mds.1:2197 currently dispatched 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: slow request 208.301405 seconds old, received at 2023-08-25T02:28:06.091009+0000: peer_request:mds.1:2229 currently dispatched 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: slow request 208.294202 seconds old, received at 2023-08-25T02:28:06.098212+0000: peer_request:mds.1:2261 currently dispatched 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: slow request 208.286964 seconds old, received at 2023-08-25T02:28:06.105449+0000: peer_request:mds.1:2293 currently dispatched 2023-08-25T02:31:35.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:34 smithi089 ceph-mon[98732]: slow request 208.279733 seconds old, received at 2023-08-25T02:28:06.112681+0000: peer_request:mds.1:2325 currently dispatched 2023-08-25T02:31:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: 990 slow requests, 5 included below; oldest blocked for > 218.723235 secs 2023-08-25T02:31:35.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: slow request 217.360585 seconds old, received at 2023-08-25T02:27:56.983508+0000: client_request(mds.1:2507 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: slow request 217.329847 seconds old, received at 2023-08-25T02:27:57.014246+0000: client_request(mds.1:2539 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: slow request 217.297970 seconds old, received at 2023-08-25T02:27:57.046122+0000: client_request(mds.1:2571 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: slow request 217.267654 seconds old, received at 2023-08-25T02:27:57.076439+0000: client_request(mds.1:2603 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: slow request 217.237873 seconds old, received at 2023-08-25T02:27:57.106220+0000: client_request(mds.1:2635 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:35 smithi121 ceph-mon[100099]: pgmap v155: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.7 MiB/s wr, 10 op/s 2023-08-25T02:31:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: 990 slow requests, 5 included below; oldest blocked for > 218.723235 secs 2023-08-25T02:31:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: slow request 217.360585 seconds old, received at 2023-08-25T02:27:56.983508+0000: client_request(mds.1:2507 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: slow request 217.329847 seconds old, received at 2023-08-25T02:27:57.014246+0000: client_request(mds.1:2539 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: slow request 217.297970 seconds old, received at 2023-08-25T02:27:57.046122+0000: client_request(mds.1:2571 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: slow request 217.267654 seconds old, received at 2023-08-25T02:27:57.076439+0000: client_request(mds.1:2603 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: slow request 217.237873 seconds old, received at 2023-08-25T02:27:57.106220+0000: client_request(mds.1:2635 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:35.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:35 smithi122 ceph-mon[100457]: pgmap v155: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.7 MiB/s wr, 10 op/s 2023-08-25T02:31:36.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: 990 slow requests, 5 included below; oldest blocked for > 218.723235 secs 2023-08-25T02:31:36.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: slow request 217.360585 seconds old, received at 2023-08-25T02:27:56.983508+0000: client_request(mds.1:2507 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:36.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: slow request 217.329847 seconds old, received at 2023-08-25T02:27:57.014246+0000: client_request(mds.1:2539 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:36.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: slow request 217.297970 seconds old, received at 2023-08-25T02:27:57.046122+0000: client_request(mds.1:2571 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:36.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: slow request 217.267654 seconds old, received at 2023-08-25T02:27:57.076439+0000: client_request(mds.1:2603 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:36.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: slow request 217.237873 seconds old, received at 2023-08-25T02:27:57.106220+0000: client_request(mds.1:2635 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:36.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:35 smithi089 ceph-mon[98732]: pgmap v155: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.7 MiB/s wr, 10 op/s 2023-08-25T02:31:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:37 smithi121 ceph-mon[100099]: pgmap v156: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.0 MiB/s wr, 16 op/s 2023-08-25T02:31:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:37 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:37 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:37 smithi122 ceph-mon[100457]: pgmap v156: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.0 MiB/s wr, 16 op/s 2023-08-25T02:31:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:37 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:37 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:38.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:37 smithi089 ceph-mon[98732]: pgmap v156: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.0 MiB/s wr, 16 op/s 2023-08-25T02:31:38.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:37 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:38.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:37 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:38.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:38 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:38 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:38 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:38 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:39.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:38 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:39.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:38 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: pgmap v157: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: 1859 slow requests, 5 included below; oldest blocked for > 213.562178 secs 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: slow request 213.272570 seconds old, received at 2023-08-25T02:28:06.119914+0000: peer_request:mds.1:2357 currently dispatched 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: slow request 213.265389 seconds old, received at 2023-08-25T02:28:06.127095+0000: peer_request:mds.1:2389 currently dispatched 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: slow request 213.258095 seconds old, received at 2023-08-25T02:28:06.134389+0000: peer_request:mds.1:2421 currently dispatched 2023-08-25T02:31:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: slow request 213.250822 seconds old, received at 2023-08-25T02:28:06.141662+0000: peer_request:mds.1:2453 currently dispatched 2023-08-25T02:31:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: slow request 213.243599 seconds old, received at 2023-08-25T02:28:06.148885+0000: peer_request:mds.1:2485 currently dispatched 2023-08-25T02:31:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:31:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:31:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:39 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:31:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: pgmap v157: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:31:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: 1859 slow requests, 5 included below; oldest blocked for > 213.562178 secs 2023-08-25T02:31:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: slow request 213.272570 seconds old, received at 2023-08-25T02:28:06.119914+0000: peer_request:mds.1:2357 currently dispatched 2023-08-25T02:31:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: slow request 213.265389 seconds old, received at 2023-08-25T02:28:06.127095+0000: peer_request:mds.1:2389 currently dispatched 2023-08-25T02:31:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: slow request 213.258095 seconds old, received at 2023-08-25T02:28:06.134389+0000: peer_request:mds.1:2421 currently dispatched 2023-08-25T02:31:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: slow request 213.250822 seconds old, received at 2023-08-25T02:28:06.141662+0000: peer_request:mds.1:2453 currently dispatched 2023-08-25T02:31:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: slow request 213.243599 seconds old, received at 2023-08-25T02:28:06.148885+0000: peer_request:mds.1:2485 currently dispatched 2023-08-25T02:31:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:31:39.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:31:39.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:39 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:31:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 -- 172.21.15.89:0/2067116182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8414100440 msgr2=0x7f841410c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 --2- 172.21.15.89:0/2067116182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8414100440 0x7f841410c650 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f84040099c0 tx=0x7f8404009cd0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 -- 172.21.15.89:0/2067116182 shutdown_connections 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 --2- 172.21.15.89:0/2067116182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8414100440 0x7f841410c650 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 --2- 172.21.15.89:0/2067116182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414105840 0x7f8414105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 --2- 172.21.15.89:0/2067116182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84140ffa80 0x7f84140fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 -- 172.21.15.89:0/2067116182 >> 172.21.15.89:0/2067116182 conn(0x7f8414075250 msgr2=0x7f8414075660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 -- 172.21.15.89:0/2067116182 shutdown_connections 2023-08-25T02:31:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.001+0000 7f8419028700 1 -- 172.21.15.89:0/2067116182 wait complete. 2023-08-25T02:31:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 Processor -- start 2023-08-25T02:31:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 -- start start 2023-08-25T02:31:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 0x7f8414197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8414105840 0x7f84141a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a2d40 con 0x7f84140ffa80 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a2eb0 con 0x7f8414105840 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8419028700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a3070 con 0x7f8414100440 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8412d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8412d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49308/0 (socket says 172.21.15.89:49308) 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f841259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 0x7f8414197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f8412d9d700 1 -- 172.21.15.89:0/961774126 learned_addr learned my addr 172.21.15.89:0/961774126 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f841359e700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8414105840 0x7f84141a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f841259c700 1 -- 172.21.15.89:0/961774126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8414105840 msgr2=0x7f84141a2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f841259c700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8414105840 0x7f84141a2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.002+0000 7f841259c700 1 -- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 msgr2=0x7f84141978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f841259c700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f841259c700 1 -- 172.21.15.89:0/961774126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040095f0 con 0x7f8414100440 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f8412d9d700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f841259c700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 0x7f8414197e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f840800e9f0 tx=0x7f840800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f83fbfff700 1 -- 172.21.15.89:0/961774126 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800c9c0 con 0x7f8414100440 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84141a3350 con 0x7f8414100440 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f83fbfff700 1 -- 172.21.15.89:0/961774126 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8408010430 con 0x7f8414100440 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f83fbfff700 1 -- 172.21.15.89:0/961774126 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408018680 con 0x7f8414100440 2023-08-25T02:31:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.003+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141a38b0 con 0x7f8414100440 2023-08-25T02:31:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.005+0000 7f83fbfff700 1 -- 172.21.15.89:0/961774126 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f8408010860 con 0x7f8414100440 2023-08-25T02:31:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f83fbfff700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 0x7f83fc0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f8412d9d700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 0x7f83fc0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f83fbfff700 1 -- 172.21.15.89:0/961774126 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f8408014070 con 0x7f8414100440 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f8412d9d700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 0x7f83fc0a2580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f840c009c10 tx=0x7f840c00b040 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f84251ccb80 1 Processor -- start 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.006+0000 7f84251ccb80 1 -- start start 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b81e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b86850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54ae5750 con 0x560b54b82370 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54ad6720 con 0x560b54b863f0 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f84251ccb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54ad4040 con 0x560b54b81a50 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49318/0 (socket says 172.21.15.89:49318) 2023-08-25T02:31:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 -- 172.21.15.89:0/2200985946 learned_addr learned my addr 172.21.15.89:0/2200985946 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b81e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841359e700 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b86850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 -- 172.21.15.89:0/2200985946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 msgr2=0x560b54b81e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b81e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 -- 172.21.15.89:0/2200985946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 msgr2=0x560b54b86850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b86850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 -- 172.21.15.89:0/2200985946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b5494f5d0 con 0x560b54b82370 2023-08-25T02:31:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f841259c700 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f84080a0760 tx=0x7f84080988b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299802794f354ff0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f83f97fa700 1 -- 172.21.15.89:0/2200985946 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840809ccb0 con 0x560b54b82370 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f83f97fa700 1 -- 172.21.15.89:0/2200985946 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8408099e90 con 0x560b54b82370 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.007+0000 7f83f97fa700 1 -- 172.21.15.89:0/2200985946 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408098370 con 0x560b54b82370 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- 172.21.15.89:0/2200985946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 msgr2=0x560b54b85eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f84080a0760 tx=0x7f84080988b0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- 172.21.15.89:0/2200985946 shutdown_connections 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b86850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b81e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2200985946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b85eb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- 172.21.15.89:0/2200985946 >> 172.21.15.89:0/2200985946 conn(0x560b54af2080 msgr2=0x560b54b77d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- 172.21.15.89:0/2200985946 shutdown_connections 2023-08-25T02:31:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- 172.21.15.89:0/2200985946 wait complete. 2023-08-25T02:31:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 -- start start 2023-08-25T02:31:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.008+0000 7f84251ccb80 1 Processor -- start 2023-08-25T02:31:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 0x560b54b8adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b863f0 0x560b54b8b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54acf930 con 0x560b54b81a50 2023-08-25T02:31:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54a6db50 con 0x560b54b82370 2023-08-25T02:31:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54a6d850 con 0x560b54b863f0 2023-08-25T02:31:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 10 client.0 _ll_register_callbacks cb 0x560b549b7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49322/0 (socket says 172.21.15.89:49322) 2023-08-25T02:31:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 -- 172.21.15.89:0/2878568852 learned_addr learned my addr 172.21.15.89:0/2878568852 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f841359e700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b863f0 0x560b54b8b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f841259c700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 0x560b54b8adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 -- 172.21.15.89:0/2878568852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b863f0 msgr2=0x560b54b8b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b863f0 0x560b54b8b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 -- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 msgr2=0x560b54b8adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 0x560b54b8adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 -- 172.21.15.89:0/2878568852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141a38b0 con 0x560b54b81a50 2023-08-25T02:31:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f841259c700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 0x560b54b8adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:31:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f840c015700 tx=0x7f840c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9348f01665f34ca0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f83f6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c017850 con 0x560b54b81a50 2023-08-25T02:31:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f840c0179b0 con 0x560b54b81a50 2023-08-25T02:31:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.009+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b54b4d160 con 0x560b54b81a50 2023-08-25T02:31:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f840c017b70 con 0x560b54b81a50 2023-08-25T02:31:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 10 client.15735 fetch_fsmap learned FSMap version 46 2023-08-25T02:31:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) v3 -- 0x560b548fbd80 con 0x560b54b81a50 2023-08-25T02:31:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560b54b4d160 con 0x560b54b81a50 2023-08-25T02:31:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 46) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f840c023c90 con 0x560b54b81a50 2023-08-25T02:31:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 10 client.15735 fetch_fsmap finished waiting for FSMap version 46 2023-08-25T02:31:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 20 client.15735 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54b9b620 0x560b54b9da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 4 client.15735 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:31:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b54a75b40 con 0x560b54b9b620 2023-08-25T02:31:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.010+0000 7f841259c700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54b9b620 0x560b54b9da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f83f6ff5700 10 client.15735 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f841259c700 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54b9b620 0x560b54b9da40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4827c2c1b428a10b server_cookie=56d6626bce5716e4 in_seq=0 out_seq=0 2023-08-25T02:31:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2878568852 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560b54a75b40 con 0x560b54b9b620 2023-08-25T02:31:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f83f6ff5700 10 client.15735 handle_command_reply: tid=0 2023-08-25T02:31:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 client.15735 shutdown 2023-08-25T02:31:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 msgr2=0x560b54b8dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f840c015700 tx=0x7f840c004ad0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 shutdown_connections 2023-08-25T02:31:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54b9b620 0x560b54b9da40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b863f0 0x560b54b8b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b81a50 0x560b54b8dbb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f83f6ff5700 0 client.15735 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2878568852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b82370 0x560b54b8adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 >> 172.21.15.89:0/2878568852 conn(0x560b54af2080 msgr2=0x560b54b77ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.011+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 shutdown_connections 2023-08-25T02:31:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.012+0000 7f84251ccb80 1 -- 172.21.15.89:0/2878568852 wait complete. 2023-08-25T02:31:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.012+0000 7f84251ccb80 20 client.15735 trim_cache size 0 max 16384 2023-08-25T02:31:40.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 Processor -- start 2023-08-25T02:31:40.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 -- start start 2023-08-25T02:31:40.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b84400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b549518c0 con 0x560b54b82370 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54b98c60 con 0x560b54b863f0 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f84251ccb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54b78c60 con 0x560b54b81a50 2023-08-25T02:31:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49332/0 (socket says 172.21.15.89:49332) 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f841259c700 1 -- 172.21.15.89:0/3311349101 learned_addr learned my addr 172.21.15.89:0/3311349101 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.026+0000 7f8412d9d700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841359e700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b84400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 -- 172.21.15.89:0/3311349101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 msgr2=0x560b54b8d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 -- 172.21.15.89:0/3311349101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 msgr2=0x560b54b84400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b84400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 -- 172.21.15.89:0/3311349101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b548e4c00 con 0x560b54b82370 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841359e700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b84400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f841259c700 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8408095380 tx=0x7f8408010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4c3380b665bc3f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f83f9ffb700 1 -- 172.21.15.89:0/3311349101 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080105a0 con 0x560b54b82370 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f83f9ffb700 1 -- 172.21.15.89:0/3311349101 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8408099e90 con 0x560b54b82370 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f83f9ffb700 1 -- 172.21.15.89:0/3311349101 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408099340 con 0x560b54b82370 2023-08-25T02:31:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 -- 172.21.15.89:0/3311349101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 msgr2=0x560b54b92660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8408095380 tx=0x7f8408010c60 comp rx=0 tx=0).stop 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 -- 172.21.15.89:0/3311349101 shutdown_connections 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54b84400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 --2- 172.21.15.89:0/3311349101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b92660 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.027+0000 7f84251ccb80 1 -- 172.21.15.89:0/3311349101 >> 172.21.15.89:0/3311349101 conn(0x560b54af2080 msgr2=0x560b54b887e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- 172.21.15.89:0/3311349101 shutdown_connections 2023-08-25T02:31:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- 172.21.15.89:0/3311349101 wait complete. 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- start start 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 Processor -- start 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54bee460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54bee9a0 con 0x560b54b82370 2023-08-25T02:31:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54beeae0 con 0x560b54b863f0 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b54beec20 con 0x560b54b81a50 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f84251ccb80 10 client.0 _ll_register_callbacks cb 0x7f840c00eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49338/0 (socket says 172.21.15.89:49338) 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 -- 172.21.15.89:0/2875303091 learned_addr learned my addr 172.21.15.89:0/2875303091 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841359e700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54bee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 -- 172.21.15.89:0/2875303091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 msgr2=0x560b54b8e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 -- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 msgr2=0x560b54bee460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.028+0000 7f841259c700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54bee460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f841259c700 1 -- 172.21.15.89:0/2875303091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b5494f5d0 con 0x560b54b82370 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f841259c700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f84080a0760 tx=0x7f84080c8f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1afcaa551870e63 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080b80c0 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f84080b8220 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b54b70600 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84080990e0 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 10 client.15741 fetch_fsmap learned FSMap version 46 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) v3 -- 0x560b549221f0 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84080c6070 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 46) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f84080a0800 con 0x560b54b82370 2023-08-25T02:31:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 10 client.15741 fetch_fsmap finished waiting for FSMap version 46 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 20 client.15741 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54befec0 0x560b54bf22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 4 client.15741 mds_command: new command op to 24509 tid=0[{"prefix": "flush journal"}] 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.029+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560b54ac8910 con 0x560b54befec0 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.030+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54befec0 0x560b54bf22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.030+0000 7f83f6ff5700 10 client.15741 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.030+0000 7f8412d9d700 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54befec0 0x560b54bf22e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e670f45e6c8f07 server_cookie=41abcd4de5b98c9 in_seq=0 out_seq=0 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.037+0000 7f83f6ff5700 1 -- 172.21.15.89:0/2875303091 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560b54ac8910 con 0x560b54befec0 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.037+0000 7f83f6ff5700 10 client.15741 handle_command_reply: tid=0 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.037+0000 7f84251ccb80 1 client.15741 shutdown 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 msgr2=0x560b54b8ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f84080a0760 tx=0x7f84080c8f70 comp rx=0 tx=0).stop 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 shutdown_connections 2023-08-25T02:31:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x560b54befec0 0x560b54bf22e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560b54b863f0 0x560b54bee460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f83f6ff5700 0 client.15741 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b54b81a50 0x560b54b8e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 --2- 172.21.15.89:0/2875303091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560b54b82370 0x560b54b8ed70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 >> 172.21.15.89:0/2875303091 conn(0x560b54af2080 msgr2=0x560b54b887e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 shutdown_connections 2023-08-25T02:31:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 1 -- 172.21.15.89:0/2875303091 wait complete. 2023-08-25T02:31:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f84251ccb80 20 client.15741 trim_cache size 0 max 16384 2023-08-25T02:31:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2023-08-25T02:31:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2023-08-25T02:31:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2023-08-25T02:31:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 msgr2=0x7f83fc0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 0x7f83fc0a2580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f840c009c10 tx=0x7f840c00b040 comp rx=0 tx=0).stop 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 msgr2=0x7f8414197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 0x7f8414197e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f840800e9f0 tx=0x7f840800ed00 comp rx=0 tx=0).stop 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 shutdown_connections 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f83fc0a00c0 0x7f83fc0a2580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8414105840 0x7f84141a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84140ffa80 0x7f84141978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 --2- 172.21.15.89:0/961774126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8414100440 0x7f8414197e10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 >> 172.21.15.89:0/961774126 conn(0x7f8414075250 msgr2=0x7f84140fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 shutdown_connections 2023-08-25T02:31:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:40.038+0000 7f8419028700 1 -- 172.21.15.89:0/961774126 wait complete. 2023-08-25T02:31:40.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: pgmap v157: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:40.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:31:40.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: 1859 slow requests, 5 included below; oldest blocked for > 213.562178 secs 2023-08-25T02:31:40.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: slow request 213.272570 seconds old, received at 2023-08-25T02:28:06.119914+0000: peer_request:mds.1:2357 currently dispatched 2023-08-25T02:31:40.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: slow request 213.265389 seconds old, received at 2023-08-25T02:28:06.127095+0000: peer_request:mds.1:2389 currently dispatched 2023-08-25T02:31:40.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: slow request 213.258095 seconds old, received at 2023-08-25T02:28:06.134389+0000: peer_request:mds.1:2421 currently dispatched 2023-08-25T02:31:40.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: slow request 213.250822 seconds old, received at 2023-08-25T02:28:06.141662+0000: peer_request:mds.1:2453 currently dispatched 2023-08-25T02:31:40.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: slow request 213.243599 seconds old, received at 2023-08-25T02:28:06.148885+0000: peer_request:mds.1:2485 currently dispatched 2023-08-25T02:31:40.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:31:40.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:31:40.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:39 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:31:40.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: 1161 slow requests, 5 included below; oldest blocked for > 223.738695 secs 2023-08-25T02:31:40.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: slow request 222.221341 seconds old, received at 2023-08-25T02:27:57.138211+0000: client_request(mds.1:2667 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: slow request 222.189447 seconds old, received at 2023-08-25T02:27:57.170105+0000: client_request(mds.1:2699 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: slow request 222.157741 seconds old, received at 2023-08-25T02:27:57.201812+0000: client_request(mds.1:2731 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: slow request 222.125934 seconds old, received at 2023-08-25T02:27:57.233619+0000: client_request(mds.1:2763 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:40 smithi121 ceph-mon[100099]: slow request 222.094850 seconds old, received at 2023-08-25T02:27:57.264703+0000: client_request(mds.1:2795 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: 1161 slow requests, 5 included below; oldest blocked for > 223.738695 secs 2023-08-25T02:31:40.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: slow request 222.221341 seconds old, received at 2023-08-25T02:27:57.138211+0000: client_request(mds.1:2667 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: slow request 222.189447 seconds old, received at 2023-08-25T02:27:57.170105+0000: client_request(mds.1:2699 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: slow request 222.157741 seconds old, received at 2023-08-25T02:27:57.201812+0000: client_request(mds.1:2731 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: slow request 222.125934 seconds old, received at 2023-08-25T02:27:57.233619+0000: client_request(mds.1:2763 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:40.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:40 smithi122 ceph-mon[100457]: slow request 222.094850 seconds old, received at 2023-08-25T02:27:57.264703+0000: client_request(mds.1:2795 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: 1161 slow requests, 5 included below; oldest blocked for > 223.738695 secs 2023-08-25T02:31:41.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: slow request 222.221341 seconds old, received at 2023-08-25T02:27:57.138211+0000: client_request(mds.1:2667 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: slow request 222.189447 seconds old, received at 2023-08-25T02:27:57.170105+0000: client_request(mds.1:2699 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: slow request 222.157741 seconds old, received at 2023-08-25T02:27:57.201812+0000: client_request(mds.1:2731 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: slow request 222.125934 seconds old, received at 2023-08-25T02:27:57.233619+0000: client_request(mds.1:2763 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:40 smithi089 ceph-mon[98732]: slow request 222.094850 seconds old, received at 2023-08-25T02:27:57.264703+0000: client_request(mds.1:2795 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:41.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:41 smithi121 ceph-mon[100099]: pgmap v158: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:41 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:41 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:41 smithi122 ceph-mon[100457]: pgmap v158: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:41 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:41 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:42.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:41 smithi089 ceph-mon[98732]: pgmap v158: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2023-08-25T02:31:42.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:41 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:41 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:43 smithi121 ceph-mon[100099]: pgmap v159: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2023-08-25T02:31:43.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:43 smithi122 ceph-mon[100457]: pgmap v159: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2023-08-25T02:31:44.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:43 smithi089 ceph-mon[98732]: pgmap v159: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2023-08-25T02:31:44.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: 1045 slow requests, 5 included below; oldest blocked for > 218.562302 secs 2023-08-25T02:31:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: slow request 218.236469 seconds old, received at 2023-08-25T02:28:06.156139+0000: peer_request:mds.1:2517 currently dispatched 2023-08-25T02:31:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: slow request 218.229232 seconds old, received at 2023-08-25T02:28:06.163376+0000: peer_request:mds.1:2549 currently dispatched 2023-08-25T02:31:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: slow request 218.221918 seconds old, received at 2023-08-25T02:28:06.170689+0000: peer_request:mds.1:2581 currently dispatched 2023-08-25T02:31:44.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: slow request 218.215744 seconds old, received at 2023-08-25T02:28:06.176863+0000: peer_request:mds.1:2613 currently dispatched 2023-08-25T02:31:44.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:44 smithi121 ceph-mon[100099]: slow request 218.209442 seconds old, received at 2023-08-25T02:28:06.183165+0000: peer_request:mds.1:2645 currently dispatched 2023-08-25T02:31:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: 1045 slow requests, 5 included below; oldest blocked for > 218.562302 secs 2023-08-25T02:31:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: slow request 218.236469 seconds old, received at 2023-08-25T02:28:06.156139+0000: peer_request:mds.1:2517 currently dispatched 2023-08-25T02:31:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: slow request 218.229232 seconds old, received at 2023-08-25T02:28:06.163376+0000: peer_request:mds.1:2549 currently dispatched 2023-08-25T02:31:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: slow request 218.221918 seconds old, received at 2023-08-25T02:28:06.170689+0000: peer_request:mds.1:2581 currently dispatched 2023-08-25T02:31:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: slow request 218.215744 seconds old, received at 2023-08-25T02:28:06.176863+0000: peer_request:mds.1:2613 currently dispatched 2023-08-25T02:31:44.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:44 smithi122 ceph-mon[100457]: slow request 218.209442 seconds old, received at 2023-08-25T02:28:06.183165+0000: peer_request:mds.1:2645 currently dispatched 2023-08-25T02:31:45.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: 1045 slow requests, 5 included below; oldest blocked for > 218.562302 secs 2023-08-25T02:31:45.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: slow request 218.236469 seconds old, received at 2023-08-25T02:28:06.156139+0000: peer_request:mds.1:2517 currently dispatched 2023-08-25T02:31:45.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: slow request 218.229232 seconds old, received at 2023-08-25T02:28:06.163376+0000: peer_request:mds.1:2549 currently dispatched 2023-08-25T02:31:45.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: slow request 218.221918 seconds old, received at 2023-08-25T02:28:06.170689+0000: peer_request:mds.1:2581 currently dispatched 2023-08-25T02:31:45.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: slow request 218.215744 seconds old, received at 2023-08-25T02:28:06.176863+0000: peer_request:mds.1:2613 currently dispatched 2023-08-25T02:31:45.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:44 smithi089 ceph-mon[98732]: slow request 218.209442 seconds old, received at 2023-08-25T02:28:06.183165+0000: peer_request:mds.1:2645 currently dispatched 2023-08-25T02:31:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: pgmap v160: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 9 op/s 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: 232 slow requests, 5 included below; oldest blocked for > 229.067597 secs 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: slow request 227.395275 seconds old, received at 2023-08-25T02:27:57.293180+0000: client_request(mds.1:2827 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: slow request 227.366539 seconds old, received at 2023-08-25T02:27:57.321916+0000: client_request(mds.1:2859 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: slow request 227.337394 seconds old, received at 2023-08-25T02:27:57.351061+0000: client_request(mds.1:2891 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: slow request 227.307729 seconds old, received at 2023-08-25T02:27:57.380726+0000: client_request(mds.1:2923 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:45 smithi121 ceph-mon[100099]: slow request 227.275536 seconds old, received at 2023-08-25T02:27:57.412919+0000: client_request(mds.1:2955 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: pgmap v160: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 9 op/s 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: 232 slow requests, 5 included below; oldest blocked for > 229.067597 secs 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: slow request 227.395275 seconds old, received at 2023-08-25T02:27:57.293180+0000: client_request(mds.1:2827 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: slow request 227.366539 seconds old, received at 2023-08-25T02:27:57.321916+0000: client_request(mds.1:2859 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: slow request 227.337394 seconds old, received at 2023-08-25T02:27:57.351061+0000: client_request(mds.1:2891 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: slow request 227.307729 seconds old, received at 2023-08-25T02:27:57.380726+0000: client_request(mds.1:2923 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:45 smithi122 ceph-mon[100457]: slow request 227.275536 seconds old, received at 2023-08-25T02:27:57.412919+0000: client_request(mds.1:2955 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:46.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: pgmap v160: 97 pgs: 97 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 9 op/s 2023-08-25T02:31:46.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: 232 slow requests, 5 included below; oldest blocked for > 229.067597 secs 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: slow request 227.395275 seconds old, received at 2023-08-25T02:27:57.293180+0000: client_request(mds.1:2827 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: slow request 227.366539 seconds old, received at 2023-08-25T02:27:57.321916+0000: client_request(mds.1:2859 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: slow request 227.337394 seconds old, received at 2023-08-25T02:27:57.351061+0000: client_request(mds.1:2891 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: slow request 227.307729 seconds old, received at 2023-08-25T02:27:57.380726+0000: client_request(mds.1:2923 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:46.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:45 smithi089 ceph-mon[98732]: slow request 227.275536 seconds old, received at 2023-08-25T02:27:57.412919+0000: client_request(mds.1:2955 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:47 smithi121 ceph-mon[100099]: pgmap v161: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2023-08-25T02:31:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:47 smithi122 ceph-mon[100457]: pgmap v161: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2023-08-25T02:31:48.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:47 smithi089 ceph-mon[98732]: pgmap v161: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2023-08-25T02:31:48.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:48 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:48.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:48 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:48.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:48 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:48.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:48 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:49.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:48 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:49.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:48 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: pgmap v162: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: 1212 slow requests, 5 included below; oldest blocked for > 223.562414 secs 2023-08-25T02:31:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: slow request 223.203208 seconds old, received at 2023-08-25T02:28:06.189511+0000: peer_request:mds.1:2677 currently dispatched 2023-08-25T02:31:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: slow request 223.196931 seconds old, received at 2023-08-25T02:28:06.195788+0000: peer_request:mds.1:2709 currently dispatched 2023-08-25T02:31:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: slow request 223.190583 seconds old, received at 2023-08-25T02:28:06.202136+0000: peer_request:mds.1:2741 currently dispatched 2023-08-25T02:31:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: slow request 223.184204 seconds old, received at 2023-08-25T02:28:06.208515+0000: peer_request:mds.1:2773 currently dispatched 2023-08-25T02:31:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:49 smithi121 ceph-mon[100099]: slow request 223.177187 seconds old, received at 2023-08-25T02:28:06.215532+0000: peer_request:mds.1:2805 currently dispatched 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: pgmap v162: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: 1212 slow requests, 5 included below; oldest blocked for > 223.562414 secs 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: slow request 223.203208 seconds old, received at 2023-08-25T02:28:06.189511+0000: peer_request:mds.1:2677 currently dispatched 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: slow request 223.196931 seconds old, received at 2023-08-25T02:28:06.195788+0000: peer_request:mds.1:2709 currently dispatched 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: slow request 223.190583 seconds old, received at 2023-08-25T02:28:06.202136+0000: peer_request:mds.1:2741 currently dispatched 2023-08-25T02:31:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: slow request 223.184204 seconds old, received at 2023-08-25T02:28:06.208515+0000: peer_request:mds.1:2773 currently dispatched 2023-08-25T02:31:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:49 smithi122 ceph-mon[100457]: slow request 223.177187 seconds old, received at 2023-08-25T02:28:06.215532+0000: peer_request:mds.1:2805 currently dispatched 2023-08-25T02:31:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: pgmap v162: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: 1212 slow requests, 5 included below; oldest blocked for > 223.562414 secs 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: slow request 223.203208 seconds old, received at 2023-08-25T02:28:06.189511+0000: peer_request:mds.1:2677 currently dispatched 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: slow request 223.196931 seconds old, received at 2023-08-25T02:28:06.195788+0000: peer_request:mds.1:2709 currently dispatched 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: slow request 223.190583 seconds old, received at 2023-08-25T02:28:06.202136+0000: peer_request:mds.1:2741 currently dispatched 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: slow request 223.184204 seconds old, received at 2023-08-25T02:28:06.208515+0000: peer_request:mds.1:2773 currently dispatched 2023-08-25T02:31:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:49 smithi089 ceph-mon[98732]: slow request 223.177187 seconds old, received at 2023-08-25T02:28:06.215532+0000: peer_request:mds.1:2805 currently dispatched 2023-08-25T02:31:51.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: 1204 slow requests, 5 included below; oldest blocked for > 234.279466 secs 2023-08-25T02:31:51.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: slow request 232.453933 seconds old, received at 2023-08-25T02:27:57.446390+0000: client_request(mds.1:2987 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: slow request 232.423584 seconds old, received at 2023-08-25T02:27:57.476740+0000: client_request(mds.1:3019 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: slow request 232.393722 seconds old, received at 2023-08-25T02:27:57.506602+0000: client_request(mds.1:3051 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: slow request 232.364995 seconds old, received at 2023-08-25T02:27:57.535329+0000: client_request(mds.1:3083 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: slow request 232.335887 seconds old, received at 2023-08-25T02:27:57.564437+0000: client_request(mds.1:3115 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:51 smithi121 ceph-mon[100099]: pgmap v163: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: 1204 slow requests, 5 included below; oldest blocked for > 234.279466 secs 2023-08-25T02:31:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: slow request 232.453933 seconds old, received at 2023-08-25T02:27:57.446390+0000: client_request(mds.1:2987 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: slow request 232.423584 seconds old, received at 2023-08-25T02:27:57.476740+0000: client_request(mds.1:3019 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: slow request 232.393722 seconds old, received at 2023-08-25T02:27:57.506602+0000: client_request(mds.1:3051 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: slow request 232.364995 seconds old, received at 2023-08-25T02:27:57.535329+0000: client_request(mds.1:3083 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: slow request 232.335887 seconds old, received at 2023-08-25T02:27:57.564437+0000: client_request(mds.1:3115 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:51 smithi122 ceph-mon[100457]: pgmap v163: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: 1204 slow requests, 5 included below; oldest blocked for > 234.279466 secs 2023-08-25T02:31:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: slow request 232.453933 seconds old, received at 2023-08-25T02:27:57.446390+0000: client_request(mds.1:2987 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: slow request 232.423584 seconds old, received at 2023-08-25T02:27:57.476740+0000: client_request(mds.1:3019 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: slow request 232.393722 seconds old, received at 2023-08-25T02:27:57.506602+0000: client_request(mds.1:3051 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: slow request 232.364995 seconds old, received at 2023-08-25T02:27:57.535329+0000: client_request(mds.1:3083 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: slow request 232.335887 seconds old, received at 2023-08-25T02:27:57.564437+0000: client_request(mds.1:3115 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:51.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:51 smithi089 ceph-mon[98732]: pgmap v163: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.4 MiB/s wr, 11 op/s 2023-08-25T02:31:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:44400000 of 50000000 tuples (88%) done (elapsed 312.25 s, remaining 39.38 s) 2023-08-25T02:31:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:44500000 of 50000000 tuples (89%) done (elapsed 312.36 s, remaining 38.61 s) 2023-08-25T02:31:53.075 INFO:teuthology.orchestra.run.smithi089.stderr:PQputline failed 2023-08-25T02:31:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:31:53.080 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi089 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 2023-08-25T02:31:53.160 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=039c794437cd4bdda52749a26240ac89 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi089 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 2023-08-25T02:31:53.162 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-08-25T02:31:53.174 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:31:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.146+0000 7f2dd2478700 1 -- 172.21.15.89:0/1127778993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc104b10 msgr2=0x7f2dcc10c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.146+0000 7f2dd2478700 1 --2- 172.21.15.89:0/1127778993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc104b10 0x7f2dcc10c6b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400b0a0 tx=0x7f2dc400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.146+0000 7f2dd2478700 1 -- 172.21.15.89:0/1127778993 shutdown_connections 2023-08-25T02:31:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 --2- 172.21.15.89:0/1127778993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc0ffc90 0x7f2dcc100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 --2- 172.21.15.89:0/1127778993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc104b10 0x7f2dcc10c6b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 --2- 172.21.15.89:0/1127778993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dcc100640 0x7f2dcc1043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 -- 172.21.15.89:0/1127778993 >> 172.21.15.89:0/1127778993 conn(0x7f2dcc0776d0 msgr2=0x7f2dcc077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:53.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 -- 172.21.15.89:0/1127778993 shutdown_connections 2023-08-25T02:31:53.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 -- 172.21.15.89:0/1127778993 wait complete. 2023-08-25T02:31:53.214 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 Processor -- start 2023-08-25T02:31:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 -- start start 2023-08-25T02:31:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.147+0000 7f2dd2478700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc0ffc90 0x7f2dcc198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc100640 0x7f2dcc199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dcc104b10 0x7f2dcc19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc19dfc0 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc19e130 con 0x7f2dcc100640 2023-08-25T02:31:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc19e2f0 con 0x7f2dcc104b10 2023-08-25T02:31:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc0ffc90 0x7f2dcc198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc0ffc90 0x7f2dcc198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49350/0 (socket says 172.21.15.89:49350) 2023-08-25T02:31:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1797278878 learned_addr learned my addr 172.21.15.89:0/1797278878 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dcc104b10 0x7f2dcc19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc100640 0x7f2dcc199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1797278878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dcc104b10 msgr2=0x7f2dcc19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2dcc104b10 0x7f2dcc19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1797278878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc100640 msgr2=0x7f2dcc199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc100640 0x7f2dcc199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1797278878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4009d00 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dcc0ffc90 0x7f2dcc198dd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00d950 tx=0x7f2dbc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dc97fa700 1 -- 172.21.15.89:0/1797278878 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00f890 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dc97fa700 1 -- 172.21.15.89:0/1797278878 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc010460 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 -- 172.21.15.89:0/1797278878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc19e5d0 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dc97fa700 1 -- 172.21.15.89:0/1797278878 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00e610 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.148+0000 7f2dd2478700 1 -- 172.21.15.89:0/1797278878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1a5e10 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.150+0000 7f2dc97fa700 1 -- 172.21.15.89:0/1797278878 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f2dbc0108d0 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.151+0000 7f2dc97fa700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2db40a00c0 0x7f2db40a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.151+0000 7f2dc97fa700 1 -- 172.21.15.89:0/1797278878 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2dbc0cb410 con 0x7f2dcc0ffc90 2023-08-25T02:31:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.151+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2db40a00c0 0x7f2db40a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.151+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/1797278878 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2db40a00c0 0x7f2db40a2580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc19a5f0 tx=0x7f2dc0009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 Processor -- start 2023-08-25T02:31:53.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 -- start start 2023-08-25T02:31:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da35320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da2f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d997750 con 0x559f4da2f990 2023-08-25T02:31:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d988720 con 0x559f4da35860 2023-08-25T02:31:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d986040 con 0x559f4da34f40 2023-08-25T02:31:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dd0a15700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da35320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dd0a15700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49362/0 (socket says 172.21.15.89:49362) 2023-08-25T02:31:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dd0a15700 1 -- 172.21.15.89:0/4229093587 learned_addr learned my addr 172.21.15.89:0/4229093587 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da2f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.152+0000 7f2dd0a15700 1 -- 172.21.15.89:0/4229093587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 msgr2=0x559f4da35320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da35320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dd0a15700 1 -- 172.21.15.89:0/4229093587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 msgr2=0x559f4da2f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da2f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dd0a15700 1 -- 172.21.15.89:0/4229093587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f4d8015d0 con 0x559f4da2f990 2023-08-25T02:31:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009390 tx=0x7f2dc4009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a18d5f6e0dd297 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2db27fc700 1 -- 172.21.15.89:0/4229093587 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400e040 con 0x559f4da2f990 2023-08-25T02:31:53.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2db27fc700 1 -- 172.21.15.89:0/4229093587 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dc4006020 con 0x559f4da2f990 2023-08-25T02:31:53.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2db27fc700 1 -- 172.21.15.89:0/4229093587 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4003c60 con 0x559f4da2f990 2023-08-25T02:31:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dde61cb80 1 -- 172.21.15.89:0/4229093587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 msgr2=0x559f4da2fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.153+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009390 tx=0x7f2dc4009c90 comp rx=0 tx=0).stop 2023-08-25T02:31:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- 172.21.15.89:0/4229093587 shutdown_connections 2023-08-25T02:31:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da2fdd0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da35320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/4229093587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da2f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- 172.21.15.89:0/4229093587 >> 172.21.15.89:0/4229093587 conn(0x559f4d9a4080 msgr2=0x559f4da29d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- 172.21.15.89:0/4229093587 shutdown_connections 2023-08-25T02:31:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- 172.21.15.89:0/4229093587 wait complete. 2023-08-25T02:31:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- start start 2023-08-25T02:31:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 Processor -- start 2023-08-25T02:31:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da2f990 0x559f4da3b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da38f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d981930 con 0x559f4da34f40 2023-08-25T02:31:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d91fb50 con 0x559f4da35860 2023-08-25T02:31:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d91f850 con 0x559f4da2f990 2023-08-25T02:31:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dde61cb80 10 client.0 _ll_register_callbacks cb 0x559f4d869a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49366/0 (socket says 172.21.15.89:49366) 2023-08-25T02:31:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da2f990 0x559f4da3b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.154+0000 7f2dcb7fe700 1 -- 172.21.15.89:0/2809764663 learned_addr learned my addr 172.21.15.89:0/2809764663 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da38f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 -- 172.21.15.89:0/2809764663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da2f990 msgr2=0x559f4da3b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da2f990 0x559f4da3b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 -- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 msgr2=0x559f4da38f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da38f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 -- 172.21.15.89:0/2809764663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc10cf20 con 0x559f4da34f40 2023-08-25T02:31:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2dc001e710 tx=0x7f2dc00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e492c3732c7414d6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0004030 con 0x559f4da34f40 2023-08-25T02:31:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2dc0004190 con 0x559f4da34f40 2023-08-25T02:31:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f4d9ff160 con 0x559f4da34f40 2023-08-25T02:31:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dc0004350 con 0x559f4da34f40 2023-08-25T02:31:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dde61cb80 10 client.15753 fetch_fsmap learned FSMap version 50 2023-08-25T02:31:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) v3 -- 0x559f4d7add80 con 0x559f4da34f40 2023-08-25T02:31:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.155+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559f4d9ff160 con 0x559f4da34f40 2023-08-25T02:31:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 50) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2dc0031550 con 0x559f4da34f40 2023-08-25T02:31:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 10 client.15753 fetch_fsmap finished waiting for FSMap version 50 2023-08-25T02:31:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-08-25T02:31:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 20 client.15753 populate_metadata read hostname 'smithi089' 2023-08-25T02:31:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f4da491d0 0x559f4da4b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 4 client.15753 mds_command: new command op to 24509 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T02:31:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 --> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f4d927b40 con 0x559f4da491d0 2023-08-25T02:31:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f4da491d0 0x559f4da4b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dab7fe700 10 client.15753 ms_handle_connect on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.156+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f4da491d0 0x559f4da4b5f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9067eab96b2925ca server_cookie=50e61def23d74daa in_seq=0 out_seq=0 2023-08-25T02:31:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dab7fe700 1 -- 172.21.15.89:0/2809764663 <== mds.0 v2:172.21.15.121:6838/1864669733 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559f4d927b40 con 0x559f4da491d0 2023-08-25T02:31:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dab7fe700 10 client.15753 handle_command_reply: tid=0 2023-08-25T02:31:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 client.15753 shutdown 2023-08-25T02:31:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 msgr2=0x559f4da38930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2dc001e710 tx=0x7f2dc00047c0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 shutdown_connections 2023-08-25T02:31:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] conn(0x559f4da491d0 0x559f4da4b5f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da35860 0x559f4da38f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da2f990 0x559f4da3b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2809764663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da34f40 0x559f4da38930 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 >> 172.21.15.89:0/2809764663 conn(0x559f4d9a4080 msgr2=0x559f4da24820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dab7fe700 0 client.15753 ms_handle_reset on v2:172.21.15.121:6838/1864669733 2023-08-25T02:31:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 shutdown_connections 2023-08-25T02:31:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2809764663 wait complete. 2023-08-25T02:31:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.157+0000 7f2dde61cb80 20 client.15753 trim_cache size 0 max 16384 2023-08-25T02:31:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 Processor -- start 2023-08-25T02:31:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 -- start start 2023-08-25T02:31:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da2f990 0x559f4da2f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4d8038c0 con 0x559f4da35860 2023-08-25T02:31:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4da39970 con 0x559f4da2f990 2023-08-25T02:31:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4da26990 con 0x559f4da34f40 2023-08-25T02:31:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49376/0 (socket says 172.21.15.89:49376) 2023-08-25T02:31:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 -- 172.21.15.89:0/2601715081 learned_addr learned my addr 172.21.15.89:0/2601715081 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da2f990 0x559f4da2f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 -- 172.21.15.89:0/2601715081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 msgr2=0x559f4da401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 -- 172.21.15.89:0/2601715081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da2f990 msgr2=0x559f4da2f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da2f990 0x559f4da2f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 -- 172.21.15.89:0/2601715081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f4d796c00 con 0x559f4da35860 2023-08-25T02:31:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009390 tx=0x7f2dc400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b936147c3640fe1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2db2ffd700 1 -- 172.21.15.89:0/2601715081 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400e040 con 0x559f4da35860 2023-08-25T02:31:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2db2ffd700 1 -- 172.21.15.89:0/2601715081 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dc4007bc0 con 0x559f4da35860 2023-08-25T02:31:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.172+0000 7f2db2ffd700 1 -- 172.21.15.89:0/2601715081 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4003c40 con 0x559f4da35860 2023-08-25T02:31:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2601715081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 msgr2=0x559f4da44940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:31:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009390 tx=0x7f2dc400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2601715081 shutdown_connections 2023-08-25T02:31:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da2f990 0x559f4da2f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da35860 0x559f4da44940 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 --2- 172.21.15.89:0/2601715081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da34f40 0x559f4da401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2601715081 >> 172.21.15.89:0/2601715081 conn(0x559f4d9a4080 msgr2=0x559f4da4b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:31:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2601715081 shutdown_connections 2023-08-25T02:31:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- 172.21.15.89:0/2601715081 wait complete. 2023-08-25T02:31:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 -- start start 2023-08-25T02:31:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.173+0000 7f2dde61cb80 1 Processor -- start 2023-08-25T02:31:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da34f40 0x559f4da30880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da35860 0x559f4da30dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:31:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4da313a0 con 0x559f4da2f990 2023-08-25T02:31:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4da31510 con 0x559f4da34f40 2023-08-25T02:31:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4da316d0 con 0x559f4da35860 2023-08-25T02:31:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 10 client.0 _ll_register_callbacks cb 0x7f2dc0011e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T02:31:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da401f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49378/0 (socket says 172.21.15.89:49378) 2023-08-25T02:31:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1561270922 learned_addr learned my addr 172.21.15.89:0/1561270922 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:31:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dd0a15700 1 --2- 172.21.15.89:0/1561270922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da35860 0x559f4da30dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcb7fe700 1 --2- 172.21.15.89:0/1561270922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da34f40 0x559f4da30880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:31:53.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1561270922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da35860 msgr2=0x559f4da30dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1561270922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f4da35860 0x559f4da30dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1561270922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da34f40 msgr2=0x559f4da30880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T02:31:53.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1561270922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559f4da34f40 0x559f4da30880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:31:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 -- 172.21.15.89:0/1561270922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f4d8015d0 con 0x559f4da2f990 2023-08-25T02:31:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dcbfff700 1 --2- 172.21.15.89:0/1561270922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f4da2f990 0x559f4da401f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0c9160 tx=0x7f2dbc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b20ccb2203ae094 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:31:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-08-25T02:31:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dab7fe700 1 -- 172.21.15.89:0/1561270922 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0c8de0 con 0x559f4da2f990 2023-08-25T02:31:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dab7fe700 1 -- 172.21.15.89:0/1561270922 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8527+0+0 (secure 0 0 0) 0x7f2dbc0c8f40 con 0x559f4da2f990 2023-08-25T02:31:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dde61cb80 1 -- 172.21.15.89:0/1561270922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f4da22600 con 0x559f4da2f990 2023-08-25T02:31:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2023-08-25T02:31:53.174+0000 7f2dab7fe700 1 -- 172.21.15.89:0/1561270922 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dbc062030 con 0x559f4da2f990 2023-08-25T02:31:53.262 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-08-25T02:31:53.272 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-08-25T02:31:53.273 INFO:teuthology.task.exec:Executing custom commands... 2023-08-25T02:31:53.273 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:31:53.274 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-08-25T02:31:53.318 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-08-25T02:31:53.329 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:53 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:53.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:53 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:53.330 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:53 smithi089 ceph-mon[98732]: pgmap v164: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 11 MiB/s wr, 20 op/s 2023-08-25T02:31:53.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:53 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:53.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:53 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:53.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:53 smithi121 ceph-mon[100099]: pgmap v164: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 11 MiB/s wr, 20 op/s 2023-08-25T02:31:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:53 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:53 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:53 smithi122 ceph-mon[100457]: pgmap v164: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 11 MiB/s wr, 20 op/s 2023-08-25T02:31:54.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:54 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:54.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:54 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:54.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:54 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:54.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:54 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:54.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:54 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:54.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:54 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:55.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: 114 slow requests, 5 included below; oldest blocked for > 228.562528 secs 2023-08-25T02:31:55.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: slow request 228.170481 seconds old, received at 2023-08-25T02:28:06.222352+0000: peer_request:mds.1:2837 currently dispatched 2023-08-25T02:31:55.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: slow request 228.164275 seconds old, received at 2023-08-25T02:28:06.228558+0000: peer_request:mds.1:2869 currently dispatched 2023-08-25T02:31:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: slow request 228.157865 seconds old, received at 2023-08-25T02:28:06.234968+0000: peer_request:mds.1:2901 currently dispatched 2023-08-25T02:31:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: slow request 228.151617 seconds old, received at 2023-08-25T02:28:06.241216+0000: peer_request:mds.1:2933 currently dispatched 2023-08-25T02:31:55.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: slow request 228.145266 seconds old, received at 2023-08-25T02:28:06.247567+0000: peer_request:mds.1:2965 currently dispatched 2023-08-25T02:31:55.471 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:55 smithi121 ceph-mon[100099]: pgmap v165: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 15 op/s 2023-08-25T02:31:55.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: 114 slow requests, 5 included below; oldest blocked for > 228.562528 secs 2023-08-25T02:31:55.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: slow request 228.170481 seconds old, received at 2023-08-25T02:28:06.222352+0000: peer_request:mds.1:2837 currently dispatched 2023-08-25T02:31:55.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: slow request 228.164275 seconds old, received at 2023-08-25T02:28:06.228558+0000: peer_request:mds.1:2869 currently dispatched 2023-08-25T02:31:55.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: slow request 228.157865 seconds old, received at 2023-08-25T02:28:06.234968+0000: peer_request:mds.1:2901 currently dispatched 2023-08-25T02:31:55.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: slow request 228.151617 seconds old, received at 2023-08-25T02:28:06.241216+0000: peer_request:mds.1:2933 currently dispatched 2023-08-25T02:31:55.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: slow request 228.145266 seconds old, received at 2023-08-25T02:28:06.247567+0000: peer_request:mds.1:2965 currently dispatched 2023-08-25T02:31:55.480 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:55 smithi122 ceph-mon[100457]: pgmap v165: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 15 op/s 2023-08-25T02:31:55.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: 114 slow requests, 5 included below; oldest blocked for > 228.562528 secs 2023-08-25T02:31:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: slow request 228.170481 seconds old, received at 2023-08-25T02:28:06.222352+0000: peer_request:mds.1:2837 currently dispatched 2023-08-25T02:31:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: slow request 228.164275 seconds old, received at 2023-08-25T02:28:06.228558+0000: peer_request:mds.1:2869 currently dispatched 2023-08-25T02:31:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: slow request 228.157865 seconds old, received at 2023-08-25T02:28:06.234968+0000: peer_request:mds.1:2901 currently dispatched 2023-08-25T02:31:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: slow request 228.151617 seconds old, received at 2023-08-25T02:28:06.241216+0000: peer_request:mds.1:2933 currently dispatched 2023-08-25T02:31:55.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: slow request 228.145266 seconds old, received at 2023-08-25T02:28:06.247567+0000: peer_request:mds.1:2965 currently dispatched 2023-08-25T02:31:55.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:55 smithi089 ceph-mon[98732]: pgmap v165: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 15 op/s 2023-08-25T02:31:56.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: 1490 slow requests, 5 included below; oldest blocked for > 239.493962 secs 2023-08-25T02:31:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: slow request 237.521461 seconds old, received at 2023-08-25T02:27:57.593358+0000: client_request(mds.1:3147 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: slow request 237.488680 seconds old, received at 2023-08-25T02:27:57.626140+0000: client_request(mds.1:3179 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: slow request 237.455472 seconds old, received at 2023-08-25T02:27:57.659347+0000: client_request(mds.1:3211 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: slow request 237.426193 seconds old, received at 2023-08-25T02:27:57.688627+0000: client_request(mds.1:3243 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:56 smithi121 ceph-mon[100099]: slow request 237.397418 seconds old, received at 2023-08-25T02:27:57.717402+0000: client_request(mds.1:3275 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: 1490 slow requests, 5 included below; oldest blocked for > 239.493962 secs 2023-08-25T02:31:56.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: slow request 237.521461 seconds old, received at 2023-08-25T02:27:57.593358+0000: client_request(mds.1:3147 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: slow request 237.488680 seconds old, received at 2023-08-25T02:27:57.626140+0000: client_request(mds.1:3179 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: slow request 237.455472 seconds old, received at 2023-08-25T02:27:57.659347+0000: client_request(mds.1:3211 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: slow request 237.426193 seconds old, received at 2023-08-25T02:27:57.688627+0000: client_request(mds.1:3243 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:56 smithi122 ceph-mon[100457]: slow request 237.397418 seconds old, received at 2023-08-25T02:27:57.717402+0000: client_request(mds.1:3275 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: 1490 slow requests, 5 included below; oldest blocked for > 239.493962 secs 2023-08-25T02:31:56.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: slow request 237.521461 seconds old, received at 2023-08-25T02:27:57.593358+0000: client_request(mds.1:3147 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: slow request 237.488680 seconds old, received at 2023-08-25T02:27:57.626140+0000: client_request(mds.1:3179 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: slow request 237.455472 seconds old, received at 2023-08-25T02:27:57.659347+0000: client_request(mds.1:3211 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: slow request 237.426193 seconds old, received at 2023-08-25T02:27:57.688627+0000: client_request(mds.1:3243 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:56.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:56 smithi089 ceph-mon[98732]: slow request 237.397418 seconds old, received at 2023-08-25T02:27:57.717402+0000: client_request(mds.1:3275 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:31:57.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:57 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:57.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:57 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:57.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:57 smithi122 ceph-mon[100457]: pgmap v166: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 22 op/s 2023-08-25T02:31:57.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:57 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:57.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:57 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:57.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:57 smithi089 ceph-mon[98732]: pgmap v166: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 22 op/s 2023-08-25T02:31:57.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:57 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:31:57.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:57 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:57.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:57 smithi121 ceph-mon[100099]: pgmap v166: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 22 op/s 2023-08-25T02:31:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:58 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:58.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:58 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:58.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:58 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:58.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:58 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:58.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:58 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:31:58.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:58 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:31:59.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:31:59 smithi089 ceph-mon[98732]: pgmap v167: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:31:59.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:31:59 smithi121 ceph-mon[100099]: pgmap v167: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:31:59.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:31:59 smithi122 ceph-mon[100457]: pgmap v167: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:32:00.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: 1316 slow requests, 5 included below; oldest blocked for > 233.562599 secs 2023-08-25T02:32:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: slow request 233.138143 seconds old, received at 2023-08-25T02:28:06.254760+0000: peer_request:mds.1:2997 currently dispatched 2023-08-25T02:32:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: slow request 233.130938 seconds old, received at 2023-08-25T02:28:06.261965+0000: peer_request:mds.1:3029 currently dispatched 2023-08-25T02:32:00.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: slow request 233.123624 seconds old, received at 2023-08-25T02:28:06.269280+0000: peer_request:mds.1:3061 currently dispatched 2023-08-25T02:32:00.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: slow request 233.116388 seconds old, received at 2023-08-25T02:28:06.276516+0000: peer_request:mds.1:3093 currently dispatched 2023-08-25T02:32:00.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:00 smithi089 ceph-mon[98732]: slow request 233.109022 seconds old, received at 2023-08-25T02:28:06.283882+0000: peer_request:mds.1:3125 currently dispatched 2023-08-25T02:32:00.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: 1316 slow requests, 5 included below; oldest blocked for > 233.562599 secs 2023-08-25T02:32:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: slow request 233.138143 seconds old, received at 2023-08-25T02:28:06.254760+0000: peer_request:mds.1:2997 currently dispatched 2023-08-25T02:32:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: slow request 233.130938 seconds old, received at 2023-08-25T02:28:06.261965+0000: peer_request:mds.1:3029 currently dispatched 2023-08-25T02:32:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: slow request 233.123624 seconds old, received at 2023-08-25T02:28:06.269280+0000: peer_request:mds.1:3061 currently dispatched 2023-08-25T02:32:00.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: slow request 233.116388 seconds old, received at 2023-08-25T02:28:06.276516+0000: peer_request:mds.1:3093 currently dispatched 2023-08-25T02:32:00.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:00 smithi121 ceph-mon[100099]: slow request 233.109022 seconds old, received at 2023-08-25T02:28:06.283882+0000: peer_request:mds.1:3125 currently dispatched 2023-08-25T02:32:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: 1316 slow requests, 5 included below; oldest blocked for > 233.562599 secs 2023-08-25T02:32:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: slow request 233.138143 seconds old, received at 2023-08-25T02:28:06.254760+0000: peer_request:mds.1:2997 currently dispatched 2023-08-25T02:32:00.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: slow request 233.130938 seconds old, received at 2023-08-25T02:28:06.261965+0000: peer_request:mds.1:3029 currently dispatched 2023-08-25T02:32:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: slow request 233.123624 seconds old, received at 2023-08-25T02:28:06.269280+0000: peer_request:mds.1:3061 currently dispatched 2023-08-25T02:32:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: slow request 233.116388 seconds old, received at 2023-08-25T02:28:06.276516+0000: peer_request:mds.1:3093 currently dispatched 2023-08-25T02:32:00.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:00 smithi122 ceph-mon[100457]: slow request 233.109022 seconds old, received at 2023-08-25T02:28:06.283882+0000: peer_request:mds.1:3125 currently dispatched 2023-08-25T02:32:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:01 smithi089 ceph-mon[98732]: 57 slow requests, 0 included below; oldest blocked for > 244.835085 secs 2023-08-25T02:32:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:01 smithi089 ceph-mon[98732]: pgmap v168: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:32:01.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:01 smithi121 ceph-mon[100099]: 57 slow requests, 0 included below; oldest blocked for > 244.835085 secs 2023-08-25T02:32:01.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:01 smithi121 ceph-mon[100099]: pgmap v168: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:32:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:01 smithi122 ceph-mon[100457]: 57 slow requests, 0 included below; oldest blocked for > 244.835085 secs 2023-08-25T02:32:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:01 smithi122 ceph-mon[100457]: pgmap v168: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 14 op/s 2023-08-25T02:32:02.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:02 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:02.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:02 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:02.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:02 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:02.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:02 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:02.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:02 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:02.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:02 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:03.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:03 smithi089 ceph-mon[98732]: pgmap v169: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 34 op/s 2023-08-25T02:32:03.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:03 smithi121 ceph-mon[100099]: pgmap v169: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 34 op/s 2023-08-25T02:32:03.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:03 smithi122 ceph-mon[100457]: pgmap v169: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 34 op/s 2023-08-25T02:32:05.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: 56 slow requests, 4 included below; oldest blocked for > 238.562665 secs 2023-08-25T02:32:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: slow request 238.102188 seconds old, received at 2023-08-25T02:28:06.290782+0000: peer_request:mds.1:3157 currently dispatched 2023-08-25T02:32:05.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: slow request 238.096262 seconds old, received at 2023-08-25T02:28:06.296709+0000: peer_request:mds.1:3189 currently dispatched 2023-08-25T02:32:05.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: slow request 238.089860 seconds old, received at 2023-08-25T02:28:06.303110+0000: peer_request:mds.1:3221 currently dispatched 2023-08-25T02:32:05.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: slow request 238.082539 seconds old, received at 2023-08-25T02:28:06.310432+0000: peer_request:mds.1:3253 currently dispatched 2023-08-25T02:32:05.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:05 smithi089 ceph-mon[98732]: pgmap v170: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 25 op/s 2023-08-25T02:32:05.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:05.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:05.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: 56 slow requests, 4 included below; oldest blocked for > 238.562665 secs 2023-08-25T02:32:05.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: slow request 238.102188 seconds old, received at 2023-08-25T02:28:06.290782+0000: peer_request:mds.1:3157 currently dispatched 2023-08-25T02:32:05.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: slow request 238.096262 seconds old, received at 2023-08-25T02:28:06.296709+0000: peer_request:mds.1:3189 currently dispatched 2023-08-25T02:32:05.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: slow request 238.089860 seconds old, received at 2023-08-25T02:28:06.303110+0000: peer_request:mds.1:3221 currently dispatched 2023-08-25T02:32:05.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: slow request 238.082539 seconds old, received at 2023-08-25T02:28:06.310432+0000: peer_request:mds.1:3253 currently dispatched 2023-08-25T02:32:05.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:05 smithi121 ceph-mon[100099]: pgmap v170: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 25 op/s 2023-08-25T02:32:05.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:05.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:05.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: 56 slow requests, 4 included below; oldest blocked for > 238.562665 secs 2023-08-25T02:32:05.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: slow request 238.102188 seconds old, received at 2023-08-25T02:28:06.290782+0000: peer_request:mds.1:3157 currently dispatched 2023-08-25T02:32:05.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: slow request 238.096262 seconds old, received at 2023-08-25T02:28:06.296709+0000: peer_request:mds.1:3189 currently dispatched 2023-08-25T02:32:05.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: slow request 238.089860 seconds old, received at 2023-08-25T02:28:06.303110+0000: peer_request:mds.1:3221 currently dispatched 2023-08-25T02:32:05.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: slow request 238.082539 seconds old, received at 2023-08-25T02:28:06.310432+0000: peer_request:mds.1:3253 currently dispatched 2023-08-25T02:32:05.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:05 smithi122 ceph-mon[100457]: pgmap v170: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 25 op/s 2023-08-25T02:32:06.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:06.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:06.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: 959 slow requests, 5 included below; oldest blocked for > 250.073061 secs 2023-08-25T02:32:06.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: slow request 249.714811 seconds old, received at 2023-08-25T02:27:55.979107+0000: client_request(mds.1:1484 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: slow request 249.681474 seconds old, received at 2023-08-25T02:27:56.012444+0000: client_request(mds.1:1516 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: slow request 249.648701 seconds old, received at 2023-08-25T02:27:56.045217+0000: client_request(mds.1:1548 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: slow request 249.616389 seconds old, received at 2023-08-25T02:27:56.077529+0000: client_request(mds.1:1580 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:06 smithi089 ceph-mon[98732]: slow request 249.583347 seconds old, received at 2023-08-25T02:27:56.110571+0000: client_request(mds.1:1612 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: 959 slow requests, 5 included below; oldest blocked for > 250.073061 secs 2023-08-25T02:32:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: slow request 249.714811 seconds old, received at 2023-08-25T02:27:55.979107+0000: client_request(mds.1:1484 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: slow request 249.681474 seconds old, received at 2023-08-25T02:27:56.012444+0000: client_request(mds.1:1516 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: slow request 249.648701 seconds old, received at 2023-08-25T02:27:56.045217+0000: client_request(mds.1:1548 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: slow request 249.616389 seconds old, received at 2023-08-25T02:27:56.077529+0000: client_request(mds.1:1580 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:06 smithi121 ceph-mon[100099]: slow request 249.583347 seconds old, received at 2023-08-25T02:27:56.110571+0000: client_request(mds.1:1612 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:06.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:06.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: 959 slow requests, 5 included below; oldest blocked for > 250.073061 secs 2023-08-25T02:32:06.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: slow request 249.714811 seconds old, received at 2023-08-25T02:27:55.979107+0000: client_request(mds.1:1484 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: slow request 249.681474 seconds old, received at 2023-08-25T02:27:56.012444+0000: client_request(mds.1:1516 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: slow request 249.648701 seconds old, received at 2023-08-25T02:27:56.045217+0000: client_request(mds.1:1548 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: slow request 249.616389 seconds old, received at 2023-08-25T02:27:56.077529+0000: client_request(mds.1:1580 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:06.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:06 smithi122 ceph-mon[100457]: slow request 249.583347 seconds old, received at 2023-08-25T02:27:56.110571+0000: client_request(mds.1:1612 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to wrlock, waiting 2023-08-25T02:32:07.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:07 smithi089 ceph-mon[98732]: pgmap v171: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 29 op/s 2023-08-25T02:32:07.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:07 smithi121 ceph-mon[100099]: pgmap v171: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 29 op/s 2023-08-25T02:32:07.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:07 smithi122 ceph-mon[100457]: pgmap v171: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 29 op/s 2023-08-25T02:32:09.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:09 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:09.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:09 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:09.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:09 smithi089 ceph-mon[98732]: pgmap v172: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:09 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:09.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:09 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:09.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:09 smithi121 ceph-mon[100099]: pgmap v172: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:09 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:09.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:09 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:09.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:09 smithi122 ceph-mon[100457]: pgmap v172: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:10.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: 282 slow requests, 5 included below; oldest blocked for > 243.562779 secs 2023-08-25T02:32:10.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: slow request 243.474593 seconds old, received at 2023-08-25T02:28:05.918491+0000: peer_request:mds.1:1494 currently dispatched 2023-08-25T02:32:10.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: slow request 243.466891 seconds old, received at 2023-08-25T02:28:05.926193+0000: peer_request:mds.1:1526 currently dispatched 2023-08-25T02:32:10.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: slow request 243.459158 seconds old, received at 2023-08-25T02:28:05.933926+0000: peer_request:mds.1:1558 currently dispatched 2023-08-25T02:32:10.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: slow request 243.451425 seconds old, received at 2023-08-25T02:28:05.941659+0000: peer_request:mds.1:1590 currently dispatched 2023-08-25T02:32:10.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:10 smithi089 ceph-mon[98732]: slow request 243.443662 seconds old, received at 2023-08-25T02:28:05.949422+0000: peer_request:mds.1:1622 currently dispatched 2023-08-25T02:32:10.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: 282 slow requests, 5 included below; oldest blocked for > 243.562779 secs 2023-08-25T02:32:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: slow request 243.474593 seconds old, received at 2023-08-25T02:28:05.918491+0000: peer_request:mds.1:1494 currently dispatched 2023-08-25T02:32:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: slow request 243.466891 seconds old, received at 2023-08-25T02:28:05.926193+0000: peer_request:mds.1:1526 currently dispatched 2023-08-25T02:32:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: slow request 243.459158 seconds old, received at 2023-08-25T02:28:05.933926+0000: peer_request:mds.1:1558 currently dispatched 2023-08-25T02:32:10.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: slow request 243.451425 seconds old, received at 2023-08-25T02:28:05.941659+0000: peer_request:mds.1:1590 currently dispatched 2023-08-25T02:32:10.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:10 smithi121 ceph-mon[100099]: slow request 243.443662 seconds old, received at 2023-08-25T02:28:05.949422+0000: peer_request:mds.1:1622 currently dispatched 2023-08-25T02:32:10.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: 282 slow requests, 5 included below; oldest blocked for > 243.562779 secs 2023-08-25T02:32:10.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: slow request 243.474593 seconds old, received at 2023-08-25T02:28:05.918491+0000: peer_request:mds.1:1494 currently dispatched 2023-08-25T02:32:10.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: slow request 243.466891 seconds old, received at 2023-08-25T02:28:05.926193+0000: peer_request:mds.1:1526 currently dispatched 2023-08-25T02:32:10.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: slow request 243.459158 seconds old, received at 2023-08-25T02:28:05.933926+0000: peer_request:mds.1:1558 currently dispatched 2023-08-25T02:32:10.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: slow request 243.451425 seconds old, received at 2023-08-25T02:28:05.941659+0000: peer_request:mds.1:1590 currently dispatched 2023-08-25T02:32:10.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:10 smithi122 ceph-mon[100457]: slow request 243.443662 seconds old, received at 2023-08-25T02:28:05.949422+0000: peer_request:mds.1:1622 currently dispatched 2023-08-25T02:32:11.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: pgmap v173: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:11.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: 1244 slow requests, 5 included below; oldest blocked for > 255.076824 secs 2023-08-25T02:32:11.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: slow request 254.553541 seconds old, received at 2023-08-25T02:27:56.144141+0000: client_request(mds.1:1644 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: slow request 254.519876 seconds old, received at 2023-08-25T02:27:56.177805+0000: client_request(mds.1:1676 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: slow request 254.486457 seconds old, received at 2023-08-25T02:27:56.211224+0000: client_request(mds.1:1708 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: slow request 254.452944 seconds old, received at 2023-08-25T02:27:56.244737+0000: client_request(mds.1:1740 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:11 smithi089 ceph-mon[98732]: slow request 254.420164 seconds old, received at 2023-08-25T02:27:56.277517+0000: client_request(mds.1:1772 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: pgmap v173: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:11.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: 1244 slow requests, 5 included below; oldest blocked for > 255.076824 secs 2023-08-25T02:32:11.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: slow request 254.553541 seconds old, received at 2023-08-25T02:27:56.144141+0000: client_request(mds.1:1644 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: slow request 254.519876 seconds old, received at 2023-08-25T02:27:56.177805+0000: client_request(mds.1:1676 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: slow request 254.486457 seconds old, received at 2023-08-25T02:27:56.211224+0000: client_request(mds.1:1708 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: slow request 254.452944 seconds old, received at 2023-08-25T02:27:56.244737+0000: client_request(mds.1:1740 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:11 smithi121 ceph-mon[100099]: slow request 254.420164 seconds old, received at 2023-08-25T02:27:56.277517+0000: client_request(mds.1:1772 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: pgmap v173: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.4 MiB/s wr, 23 op/s 2023-08-25T02:32:11.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: 1244 slow requests, 5 included below; oldest blocked for > 255.076824 secs 2023-08-25T02:32:11.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: slow request 254.553541 seconds old, received at 2023-08-25T02:27:56.144141+0000: client_request(mds.1:1644 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: slow request 254.519876 seconds old, received at 2023-08-25T02:27:56.177805+0000: client_request(mds.1:1676 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: slow request 254.486457 seconds old, received at 2023-08-25T02:27:56.211224+0000: client_request(mds.1:1708 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: slow request 254.452944 seconds old, received at 2023-08-25T02:27:56.244737+0000: client_request(mds.1:1740 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:11.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:11 smithi122 ceph-mon[100457]: slow request 254.420164 seconds old, received at 2023-08-25T02:27:56.277517+0000: client_request(mds.1:1772 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:13.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:13 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:13.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:13 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:13.585 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:13 smithi089 ceph-mon[98732]: pgmap v174: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.4 MiB/s wr, 27 op/s 2023-08-25T02:32:13.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:13 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:13.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:13 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:13.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:13 smithi121 ceph-mon[100099]: pgmap v174: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.4 MiB/s wr, 27 op/s 2023-08-25T02:32:13.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:13 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:13.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:13 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:13.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:13 smithi122 ceph-mon[100457]: pgmap v174: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.4 MiB/s wr, 27 op/s 2023-08-25T02:32:14.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:14 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:14.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:14 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:14.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:14 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:14.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:14 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:14.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:14 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:14.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:14 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:15.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: 676 slow requests, 5 included below; oldest blocked for > 248.563498 secs 2023-08-25T02:32:15.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: slow request 248.436528 seconds old, received at 2023-08-25T02:28:05.957275+0000: peer_request:mds.1:1654 currently dispatched 2023-08-25T02:32:15.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: slow request 248.428722 seconds old, received at 2023-08-25T02:28:05.965081+0000: peer_request:mds.1:1686 currently dispatched 2023-08-25T02:32:15.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: slow request 248.420950 seconds old, received at 2023-08-25T02:28:05.972853+0000: peer_request:mds.1:1718 currently dispatched 2023-08-25T02:32:15.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: slow request 248.413299 seconds old, received at 2023-08-25T02:28:05.980504+0000: peer_request:mds.1:1750 currently dispatched 2023-08-25T02:32:15.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: slow request 248.406118 seconds old, received at 2023-08-25T02:28:05.987685+0000: peer_request:mds.1:1782 currently dispatched 2023-08-25T02:32:15.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:15 smithi089 ceph-mon[98732]: pgmap v175: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:32:15.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: 676 slow requests, 5 included below; oldest blocked for > 248.563498 secs 2023-08-25T02:32:15.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: slow request 248.436528 seconds old, received at 2023-08-25T02:28:05.957275+0000: peer_request:mds.1:1654 currently dispatched 2023-08-25T02:32:15.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: slow request 248.428722 seconds old, received at 2023-08-25T02:28:05.965081+0000: peer_request:mds.1:1686 currently dispatched 2023-08-25T02:32:15.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: slow request 248.420950 seconds old, received at 2023-08-25T02:28:05.972853+0000: peer_request:mds.1:1718 currently dispatched 2023-08-25T02:32:15.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: slow request 248.413299 seconds old, received at 2023-08-25T02:28:05.980504+0000: peer_request:mds.1:1750 currently dispatched 2023-08-25T02:32:15.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: slow request 248.406118 seconds old, received at 2023-08-25T02:28:05.987685+0000: peer_request:mds.1:1782 currently dispatched 2023-08-25T02:32:15.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:15 smithi121 ceph-mon[100099]: pgmap v175: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:32:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: 676 slow requests, 5 included below; oldest blocked for > 248.563498 secs 2023-08-25T02:32:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: slow request 248.436528 seconds old, received at 2023-08-25T02:28:05.957275+0000: peer_request:mds.1:1654 currently dispatched 2023-08-25T02:32:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: slow request 248.428722 seconds old, received at 2023-08-25T02:28:05.965081+0000: peer_request:mds.1:1686 currently dispatched 2023-08-25T02:32:15.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: slow request 248.420950 seconds old, received at 2023-08-25T02:28:05.972853+0000: peer_request:mds.1:1718 currently dispatched 2023-08-25T02:32:15.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: slow request 248.413299 seconds old, received at 2023-08-25T02:28:05.980504+0000: peer_request:mds.1:1750 currently dispatched 2023-08-25T02:32:15.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: slow request 248.406118 seconds old, received at 2023-08-25T02:28:05.987685+0000: peer_request:mds.1:1782 currently dispatched 2023-08-25T02:32:15.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:15 smithi122 ceph-mon[100457]: pgmap v175: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-08-25T02:32:16.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:16 smithi089 ceph-mon[98732]: 56 slow requests, 0 included below; oldest blocked for > 260.076891 secs 2023-08-25T02:32:16.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:16 smithi121 ceph-mon[100099]: 56 slow requests, 0 included below; oldest blocked for > 260.076891 secs 2023-08-25T02:32:16.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:16 smithi122 ceph-mon[100457]: 56 slow requests, 0 included below; oldest blocked for > 260.076891 secs 2023-08-25T02:32:17.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:17 smithi089 ceph-mon[98732]: pgmap v176: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 454 KiB/s wr, 99 op/s 2023-08-25T02:32:17.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:17 smithi121 ceph-mon[100099]: pgmap v176: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 454 KiB/s wr, 99 op/s 2023-08-25T02:32:17.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:17 smithi122 ceph-mon[100457]: pgmap v176: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 454 KiB/s wr, 99 op/s 2023-08-25T02:32:18.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:18 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:18 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:18.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:18 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:18 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:18 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:18 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:19.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:19 smithi089 ceph-mon[98732]: pgmap v177: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:19.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:19 smithi121 ceph-mon[100099]: pgmap v177: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:19.727 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:19 smithi122 ceph-mon[100457]: pgmap v177: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:20.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:20 smithi089 ceph-mon[98732]: 55 slow requests, 0 included below; oldest blocked for > 253.563615 secs 2023-08-25T02:32:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:20 smithi121 ceph-mon[100099]: 55 slow requests, 0 included below; oldest blocked for > 253.563615 secs 2023-08-25T02:32:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:20 smithi122 ceph-mon[100457]: 55 slow requests, 0 included below; oldest blocked for > 253.563615 secs 2023-08-25T02:32:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:21 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:21 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:21 smithi089 ceph-mon[98732]: pgmap v178: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:21 smithi089 ceph-mon[98732]: 56 slow requests, 0 included below; oldest blocked for > 265.109174 secs 2023-08-25T02:32:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:21 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:21 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:21 smithi121 ceph-mon[100099]: pgmap v178: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:21 smithi121 ceph-mon[100099]: 56 slow requests, 0 included below; oldest blocked for > 265.109174 secs 2023-08-25T02:32:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:21 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:21 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:21 smithi122 ceph-mon[100457]: pgmap v178: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 447 KiB/s wr, 95 op/s 2023-08-25T02:32:21.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:21 smithi122 ceph-mon[100457]: 56 slow requests, 0 included below; oldest blocked for > 265.109174 secs 2023-08-25T02:32:22.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:22 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:22 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:22 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:22 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:22.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:22 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:22.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:22 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:23.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:23 smithi121 ceph-mon[100099]: pgmap v179: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 469 KiB/s wr, 104 op/s 2023-08-25T02:32:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:23 smithi122 ceph-mon[100457]: pgmap v179: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 469 KiB/s wr, 104 op/s 2023-08-25T02:32:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:23 smithi089 ceph-mon[98732]: pgmap v179: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 469 KiB/s wr, 104 op/s 2023-08-25T02:32:24.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:24 smithi089 ceph-mon[98732]: 55 slow requests, 0 included below; oldest blocked for > 258.563760 secs 2023-08-25T02:32:24.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:24 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:24.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:24 smithi121 ceph-mon[100099]: 55 slow requests, 0 included below; oldest blocked for > 258.563760 secs 2023-08-25T02:32:24.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:24 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:24 smithi122 ceph-mon[100457]: 55 slow requests, 0 included below; oldest blocked for > 258.563760 secs 2023-08-25T02:32:24.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:24 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:25 smithi089 ceph-mon[98732]: pgmap v180: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 462 KiB/s wr, 99 op/s 2023-08-25T02:32:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:25 smithi121 ceph-mon[100099]: pgmap v180: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 462 KiB/s wr, 99 op/s 2023-08-25T02:32:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:25 smithi122 ceph-mon[100457]: pgmap v180: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 462 KiB/s wr, 99 op/s 2023-08-25T02:32:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: 112 slow requests, 5 included below; oldest blocked for > 270.278014 secs 2023-08-25T02:32:27.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: slow request 269.588527 seconds old, received at 2023-08-25T02:27:56.310344+0000: client_request(mds.1:1804 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: slow request 269.555212 seconds old, received at 2023-08-25T02:27:56.343660+0000: client_request(mds.1:1836 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: slow request 269.521994 seconds old, received at 2023-08-25T02:27:56.376877+0000: client_request(mds.1:1868 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: slow request 269.491361 seconds old, received at 2023-08-25T02:27:56.407510+0000: client_request(mds.1:1900 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: slow request 269.459912 seconds old, received at 2023-08-25T02:27:56.438960+0000: client_request(mds.1:1932 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:27 smithi089 ceph-mon[98732]: pgmap v181: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 842 KiB/s wr, 117 op/s 2023-08-25T02:32:27.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: 112 slow requests, 5 included below; oldest blocked for > 270.278014 secs 2023-08-25T02:32:27.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: slow request 269.588527 seconds old, received at 2023-08-25T02:27:56.310344+0000: client_request(mds.1:1804 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: slow request 269.555212 seconds old, received at 2023-08-25T02:27:56.343660+0000: client_request(mds.1:1836 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: slow request 269.521994 seconds old, received at 2023-08-25T02:27:56.376877+0000: client_request(mds.1:1868 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: slow request 269.491361 seconds old, received at 2023-08-25T02:27:56.407510+0000: client_request(mds.1:1900 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: slow request 269.459912 seconds old, received at 2023-08-25T02:27:56.438960+0000: client_request(mds.1:1932 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:27 smithi121 ceph-mon[100099]: pgmap v181: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 842 KiB/s wr, 117 op/s 2023-08-25T02:32:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: 112 slow requests, 5 included below; oldest blocked for > 270.278014 secs 2023-08-25T02:32:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: slow request 269.588527 seconds old, received at 2023-08-25T02:27:56.310344+0000: client_request(mds.1:1804 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: slow request 269.555212 seconds old, received at 2023-08-25T02:27:56.343660+0000: client_request(mds.1:1836 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: slow request 269.521994 seconds old, received at 2023-08-25T02:27:56.376877+0000: client_request(mds.1:1868 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: slow request 269.491361 seconds old, received at 2023-08-25T02:27:56.407510+0000: client_request(mds.1:1900 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: slow request 269.459912 seconds old, received at 2023-08-25T02:27:56.438960+0000: client_request(mds.1:1932 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:27.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:27 smithi122 ceph-mon[100457]: pgmap v181: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 842 KiB/s wr, 117 op/s 2023-08-25T02:32:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:29 smithi089 ceph-mon[98732]: pgmap v182: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:29.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:29 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:29.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:29 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:29.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:29 smithi089 ceph-mon[98732]: 55 slow requests, 0 included below; oldest blocked for > 263.563896 secs 2023-08-25T02:32:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:29 smithi121 ceph-mon[100099]: pgmap v182: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:29 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:29 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:29.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:29 smithi121 ceph-mon[100099]: 55 slow requests, 0 included below; oldest blocked for > 263.563896 secs 2023-08-25T02:32:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:29 smithi122 ceph-mon[100457]: pgmap v182: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:29 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:29 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:29 smithi122 ceph-mon[100457]: 55 slow requests, 0 included below; oldest blocked for > 263.563896 secs 2023-08-25T02:32:31.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:31 smithi121 ceph-mon[100099]: pgmap v183: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:31.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:31 smithi122 ceph-mon[100457]: pgmap v183: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:31 smithi089 ceph-mon[98732]: pgmap v183: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 401 KiB/s wr, 26 op/s 2023-08-25T02:32:32.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: 165 slow requests, 5 included below; oldest blocked for > 275.301900 secs 2023-08-25T02:32:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: slow request 274.452353 seconds old, received at 2023-08-25T02:27:56.470404+0000: client_request(mds.1:1964 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: slow request 274.421534 seconds old, received at 2023-08-25T02:27:56.501224+0000: client_request(mds.1:1996 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: slow request 274.392030 seconds old, received at 2023-08-25T02:27:56.530727+0000: client_request(mds.1:2028 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: slow request 274.362651 seconds old, received at 2023-08-25T02:27:56.560107+0000: client_request(mds.1:2060 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.470 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:32 smithi121 ceph-mon[100099]: slow request 274.333248 seconds old, received at 2023-08-25T02:27:56.589510+0000: client_request(mds.1:2092 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: 165 slow requests, 5 included below; oldest blocked for > 275.301900 secs 2023-08-25T02:32:32.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: slow request 274.452353 seconds old, received at 2023-08-25T02:27:56.470404+0000: client_request(mds.1:1964 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: slow request 274.421534 seconds old, received at 2023-08-25T02:27:56.501224+0000: client_request(mds.1:1996 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: slow request 274.392030 seconds old, received at 2023-08-25T02:27:56.530727+0000: client_request(mds.1:2028 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: slow request 274.362651 seconds old, received at 2023-08-25T02:27:56.560107+0000: client_request(mds.1:2060 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:32 smithi122 ceph-mon[100457]: slow request 274.333248 seconds old, received at 2023-08-25T02:27:56.589510+0000: client_request(mds.1:2092 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: 165 slow requests, 5 included below; oldest blocked for > 275.301900 secs 2023-08-25T02:32:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: slow request 274.452353 seconds old, received at 2023-08-25T02:27:56.470404+0000: client_request(mds.1:1964 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: slow request 274.421534 seconds old, received at 2023-08-25T02:27:56.501224+0000: client_request(mds.1:1996 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: slow request 274.392030 seconds old, received at 2023-08-25T02:27:56.530727+0000: client_request(mds.1:2028 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: slow request 274.362651 seconds old, received at 2023-08-25T02:27:56.560107+0000: client_request(mds.1:2060 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:32.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:32 smithi089 ceph-mon[98732]: slow request 274.333248 seconds old, received at 2023-08-25T02:27:56.589510+0000: client_request(mds.1:2092 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:33.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:33 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:33.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:33 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:33.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:33 smithi122 ceph-mon[100457]: pgmap v184: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 36 op/s 2023-08-25T02:32:33.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:33 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:33.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:33 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:33.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:33 smithi089 ceph-mon[98732]: pgmap v184: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 36 op/s 2023-08-25T02:32:33.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:33 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:33 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:33.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:33 smithi121 ceph-mon[100099]: pgmap v184: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 36 op/s 2023-08-25T02:32:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: 221 slow requests, 5 included below; oldest blocked for > 268.564048 secs 2023-08-25T02:32:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: slow request 268.399510 seconds old, received at 2023-08-25T02:28:05.994844+0000: peer_request:mds.1:1814 currently dispatched 2023-08-25T02:32:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: slow request 268.391940 seconds old, received at 2023-08-25T02:28:06.002414+0000: peer_request:mds.1:1846 currently dispatched 2023-08-25T02:32:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: slow request 268.384143 seconds old, received at 2023-08-25T02:28:06.010211+0000: peer_request:mds.1:1878 currently dispatched 2023-08-25T02:32:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: slow request 268.376417 seconds old, received at 2023-08-25T02:28:06.017937+0000: peer_request:mds.1:1910 currently dispatched 2023-08-25T02:32:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:34 smithi089 ceph-mon[98732]: slow request 268.368602 seconds old, received at 2023-08-25T02:28:06.025752+0000: peer_request:mds.1:1942 currently dispatched 2023-08-25T02:32:34.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: 221 slow requests, 5 included below; oldest blocked for > 268.564048 secs 2023-08-25T02:32:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: slow request 268.399510 seconds old, received at 2023-08-25T02:28:05.994844+0000: peer_request:mds.1:1814 currently dispatched 2023-08-25T02:32:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: slow request 268.391940 seconds old, received at 2023-08-25T02:28:06.002414+0000: peer_request:mds.1:1846 currently dispatched 2023-08-25T02:32:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: slow request 268.384143 seconds old, received at 2023-08-25T02:28:06.010211+0000: peer_request:mds.1:1878 currently dispatched 2023-08-25T02:32:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: slow request 268.376417 seconds old, received at 2023-08-25T02:28:06.017937+0000: peer_request:mds.1:1910 currently dispatched 2023-08-25T02:32:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:34 smithi121 ceph-mon[100099]: slow request 268.368602 seconds old, received at 2023-08-25T02:28:06.025752+0000: peer_request:mds.1:1942 currently dispatched 2023-08-25T02:32:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: 221 slow requests, 5 included below; oldest blocked for > 268.564048 secs 2023-08-25T02:32:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: slow request 268.399510 seconds old, received at 2023-08-25T02:28:05.994844+0000: peer_request:mds.1:1814 currently dispatched 2023-08-25T02:32:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: slow request 268.391940 seconds old, received at 2023-08-25T02:28:06.002414+0000: peer_request:mds.1:1846 currently dispatched 2023-08-25T02:32:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: slow request 268.384143 seconds old, received at 2023-08-25T02:28:06.010211+0000: peer_request:mds.1:1878 currently dispatched 2023-08-25T02:32:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: slow request 268.376417 seconds old, received at 2023-08-25T02:28:06.017937+0000: peer_request:mds.1:1910 currently dispatched 2023-08-25T02:32:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:34 smithi122 ceph-mon[100457]: slow request 268.368602 seconds old, received at 2023-08-25T02:28:06.025752+0000: peer_request:mds.1:1942 currently dispatched 2023-08-25T02:32:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:35 smithi089 ceph-mon[98732]: pgmap v185: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-08-25T02:32:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:35 smithi121 ceph-mon[100099]: pgmap v185: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-08-25T02:32:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:35 smithi122 ceph-mon[100457]: pgmap v185: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-08-25T02:32:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: 1756 slow requests, 5 included below; oldest blocked for > 280.385809 secs 2023-08-25T02:32:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: slow request 279.387388 seconds old, received at 2023-08-25T02:27:56.619279+0000: client_request(mds.1:2124 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: slow request 279.357771 seconds old, received at 2023-08-25T02:27:56.648896+0000: client_request(mds.1:2156 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: slow request 279.328486 seconds old, received at 2023-08-25T02:27:56.678181+0000: client_request(mds.1:2188 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: slow request 279.298820 seconds old, received at 2023-08-25T02:27:56.707846+0000: client_request(mds.1:2220 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: slow request 279.269506 seconds old, received at 2023-08-25T02:27:56.737160+0000: client_request(mds.1:2252 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:37 smithi089 ceph-mon[98732]: pgmap v186: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 54 op/s 2023-08-25T02:32:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: 1756 slow requests, 5 included below; oldest blocked for > 280.385809 secs 2023-08-25T02:32:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: slow request 279.387388 seconds old, received at 2023-08-25T02:27:56.619279+0000: client_request(mds.1:2124 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: slow request 279.357771 seconds old, received at 2023-08-25T02:27:56.648896+0000: client_request(mds.1:2156 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: slow request 279.328486 seconds old, received at 2023-08-25T02:27:56.678181+0000: client_request(mds.1:2188 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: slow request 279.298820 seconds old, received at 2023-08-25T02:27:56.707846+0000: client_request(mds.1:2220 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: slow request 279.269506 seconds old, received at 2023-08-25T02:27:56.737160+0000: client_request(mds.1:2252 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:37 smithi121 ceph-mon[100099]: pgmap v186: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 54 op/s 2023-08-25T02:32:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: 1756 slow requests, 5 included below; oldest blocked for > 280.385809 secs 2023-08-25T02:32:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: slow request 279.387388 seconds old, received at 2023-08-25T02:27:56.619279+0000: client_request(mds.1:2124 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: slow request 279.357771 seconds old, received at 2023-08-25T02:27:56.648896+0000: client_request(mds.1:2156 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: slow request 279.328486 seconds old, received at 2023-08-25T02:27:56.678181+0000: client_request(mds.1:2188 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: slow request 279.298820 seconds old, received at 2023-08-25T02:27:56.707846+0000: client_request(mds.1:2220 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: slow request 279.269506 seconds old, received at 2023-08-25T02:27:56.737160+0000: client_request(mds.1:2252 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:37.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:37 smithi122 ceph-mon[100457]: pgmap v186: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 54 op/s 2023-08-25T02:32:38.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:38 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:38 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:38.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:38 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:38 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:38 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:38 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: pgmap v187: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: 929 slow requests, 5 included below; oldest blocked for > 273.564162 secs 2023-08-25T02:32:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: slow request 273.360924 seconds old, received at 2023-08-25T02:28:06.033543+0000: peer_request:mds.1:1974 currently dispatched 2023-08-25T02:32:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: slow request 273.353034 seconds old, received at 2023-08-25T02:28:06.041433+0000: peer_request:mds.1:2006 currently dispatched 2023-08-25T02:32:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: slow request 273.345762 seconds old, received at 2023-08-25T02:28:06.048705+0000: peer_request:mds.1:2038 currently dispatched 2023-08-25T02:32:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: slow request 273.338913 seconds old, received at 2023-08-25T02:28:06.055554+0000: peer_request:mds.1:2070 currently dispatched 2023-08-25T02:32:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:39 smithi089 ceph-mon[98732]: slow request 273.332032 seconds old, received at 2023-08-25T02:28:06.062435+0000: peer_request:mds.1:2102 currently dispatched 2023-08-25T02:32:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: pgmap v187: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: 929 slow requests, 5 included below; oldest blocked for > 273.564162 secs 2023-08-25T02:32:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: slow request 273.360924 seconds old, received at 2023-08-25T02:28:06.033543+0000: peer_request:mds.1:1974 currently dispatched 2023-08-25T02:32:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: slow request 273.353034 seconds old, received at 2023-08-25T02:28:06.041433+0000: peer_request:mds.1:2006 currently dispatched 2023-08-25T02:32:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: slow request 273.345762 seconds old, received at 2023-08-25T02:28:06.048705+0000: peer_request:mds.1:2038 currently dispatched 2023-08-25T02:32:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: slow request 273.338913 seconds old, received at 2023-08-25T02:28:06.055554+0000: peer_request:mds.1:2070 currently dispatched 2023-08-25T02:32:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:39 smithi121 ceph-mon[100099]: slow request 273.332032 seconds old, received at 2023-08-25T02:28:06.062435+0000: peer_request:mds.1:2102 currently dispatched 2023-08-25T02:32:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: pgmap v187: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: 929 slow requests, 5 included below; oldest blocked for > 273.564162 secs 2023-08-25T02:32:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: slow request 273.360924 seconds old, received at 2023-08-25T02:28:06.033543+0000: peer_request:mds.1:1974 currently dispatched 2023-08-25T02:32:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: slow request 273.353034 seconds old, received at 2023-08-25T02:28:06.041433+0000: peer_request:mds.1:2006 currently dispatched 2023-08-25T02:32:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: slow request 273.345762 seconds old, received at 2023-08-25T02:28:06.048705+0000: peer_request:mds.1:2038 currently dispatched 2023-08-25T02:32:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: slow request 273.338913 seconds old, received at 2023-08-25T02:28:06.055554+0000: peer_request:mds.1:2070 currently dispatched 2023-08-25T02:32:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:39 smithi122 ceph-mon[100457]: slow request 273.332032 seconds old, received at 2023-08-25T02:28:06.062435+0000: peer_request:mds.1:2102 currently dispatched 2023-08-25T02:32:40.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:40 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:32:40.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:40 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:32:41.068 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:40 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:32:41.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:41 smithi089 ceph-mon[98732]: pgmap v188: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:41.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:41 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:41.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:41 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:41.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:41 smithi121 ceph-mon[100099]: pgmap v188: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:41 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:41 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:41.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:41 smithi122 ceph-mon[100457]: pgmap v188: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 915 KiB/s wr, 36 op/s 2023-08-25T02:32:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:41 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:41 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:42.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: 1746 slow requests, 5 included below; oldest blocked for > 285.445853 secs 2023-08-25T02:32:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: slow request 284.299496 seconds old, received at 2023-08-25T02:27:56.767215+0000: client_request(mds.1:2284 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: slow request 284.270003 seconds old, received at 2023-08-25T02:27:56.796708+0000: client_request(mds.1:2316 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: slow request 284.240567 seconds old, received at 2023-08-25T02:27:56.826143+0000: client_request(mds.1:2348 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: slow request 284.209616 seconds old, received at 2023-08-25T02:27:56.857095+0000: client_request(mds.1:2380 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: slow request 284.177782 seconds old, received at 2023-08-25T02:27:56.888928+0000: client_request(mds.1:2412 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:42.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:42 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:42.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: 1746 slow requests, 5 included below; oldest blocked for > 285.445853 secs 2023-08-25T02:32:42.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: slow request 284.299496 seconds old, received at 2023-08-25T02:27:56.767215+0000: client_request(mds.1:2284 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: slow request 284.270003 seconds old, received at 2023-08-25T02:27:56.796708+0000: client_request(mds.1:2316 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: slow request 284.240567 seconds old, received at 2023-08-25T02:27:56.826143+0000: client_request(mds.1:2348 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: slow request 284.209616 seconds old, received at 2023-08-25T02:27:56.857095+0000: client_request(mds.1:2380 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: slow request 284.177782 seconds old, received at 2023-08-25T02:27:56.888928+0000: client_request(mds.1:2412 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:42.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:42.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:42 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:43.045 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: 1746 slow requests, 5 included below; oldest blocked for > 285.445853 secs 2023-08-25T02:32:43.045 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: slow request 284.299496 seconds old, received at 2023-08-25T02:27:56.767215+0000: client_request(mds.1:2284 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:43.046 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: slow request 284.270003 seconds old, received at 2023-08-25T02:27:56.796708+0000: client_request(mds.1:2316 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:43.046 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: slow request 284.240567 seconds old, received at 2023-08-25T02:27:56.826143+0000: client_request(mds.1:2348 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:43.046 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: slow request 284.209616 seconds old, received at 2023-08-25T02:27:56.857095+0000: client_request(mds.1:2380 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:43.046 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: slow request 284.177782 seconds old, received at 2023-08-25T02:27:56.888928+0000: client_request(mds.1:2412 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:43.047 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:43.047 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:42 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:43 smithi089 ceph-mon[98732]: pgmap v189: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 930 KiB/s wr, 44 op/s 2023-08-25T02:32:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:43 smithi121 ceph-mon[100099]: pgmap v189: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 930 KiB/s wr, 44 op/s 2023-08-25T02:32:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:43 smithi122 ceph-mon[100457]: pgmap v189: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 930 KiB/s wr, 44 op/s 2023-08-25T02:32:44.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: 1307 slow requests, 5 included below; oldest blocked for > 278.564278 secs 2023-08-25T02:32:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: slow request 278.324964 seconds old, received at 2023-08-25T02:28:06.069620+0000: peer_request:mds.1:2134 currently dispatched 2023-08-25T02:32:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: slow request 278.317713 seconds old, received at 2023-08-25T02:28:06.076871+0000: peer_request:mds.1:2166 currently dispatched 2023-08-25T02:32:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: slow request 278.310472 seconds old, received at 2023-08-25T02:28:06.084112+0000: peer_request:mds.1:2198 currently dispatched 2023-08-25T02:32:44.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: slow request 278.303357 seconds old, received at 2023-08-25T02:28:06.091227+0000: peer_request:mds.1:2230 currently dispatched 2023-08-25T02:32:44.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:44 smithi089 ceph-mon[98732]: slow request 278.296155 seconds old, received at 2023-08-25T02:28:06.098429+0000: peer_request:mds.1:2262 currently dispatched 2023-08-25T02:32:44.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: 1307 slow requests, 5 included below; oldest blocked for > 278.564278 secs 2023-08-25T02:32:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: slow request 278.324964 seconds old, received at 2023-08-25T02:28:06.069620+0000: peer_request:mds.1:2134 currently dispatched 2023-08-25T02:32:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: slow request 278.317713 seconds old, received at 2023-08-25T02:28:06.076871+0000: peer_request:mds.1:2166 currently dispatched 2023-08-25T02:32:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: slow request 278.310472 seconds old, received at 2023-08-25T02:28:06.084112+0000: peer_request:mds.1:2198 currently dispatched 2023-08-25T02:32:44.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: slow request 278.303357 seconds old, received at 2023-08-25T02:28:06.091227+0000: peer_request:mds.1:2230 currently dispatched 2023-08-25T02:32:44.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:44 smithi121 ceph-mon[100099]: slow request 278.296155 seconds old, received at 2023-08-25T02:28:06.098429+0000: peer_request:mds.1:2262 currently dispatched 2023-08-25T02:32:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: 1307 slow requests, 5 included below; oldest blocked for > 278.564278 secs 2023-08-25T02:32:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: slow request 278.324964 seconds old, received at 2023-08-25T02:28:06.069620+0000: peer_request:mds.1:2134 currently dispatched 2023-08-25T02:32:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: slow request 278.317713 seconds old, received at 2023-08-25T02:28:06.076871+0000: peer_request:mds.1:2166 currently dispatched 2023-08-25T02:32:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: slow request 278.310472 seconds old, received at 2023-08-25T02:28:06.084112+0000: peer_request:mds.1:2198 currently dispatched 2023-08-25T02:32:44.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: slow request 278.303357 seconds old, received at 2023-08-25T02:28:06.091227+0000: peer_request:mds.1:2230 currently dispatched 2023-08-25T02:32:44.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:44 smithi122 ceph-mon[100457]: slow request 278.296155 seconds old, received at 2023-08-25T02:28:06.098429+0000: peer_request:mds.1:2262 currently dispatched 2023-08-25T02:32:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: pgmap v190: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 146 KiB/s wr, 34 op/s 2023-08-25T02:32:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:32:45.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:32:45.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:45 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:32:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: pgmap v190: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 146 KiB/s wr, 34 op/s 2023-08-25T02:32:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:32:45.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:32:45.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:45 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:32:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: pgmap v190: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 146 KiB/s wr, 34 op/s 2023-08-25T02:32:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:32:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:32:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:45 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:32:46.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:46 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:46.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:46 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:46.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:46 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:46.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:46 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:46.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:46 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:46.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:46 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: 1031 slow requests, 5 included below; oldest blocked for > 290.685919 secs 2023-08-25T02:32:47.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: slow request 289.386025 seconds old, received at 2023-08-25T02:27:56.920752+0000: client_request(mds.1:2444 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: slow request 289.354085 seconds old, received at 2023-08-25T02:27:56.952691+0000: client_request(mds.1:2476 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: slow request 289.322290 seconds old, received at 2023-08-25T02:27:56.984486+0000: client_request(mds.1:2508 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: slow request 289.291548 seconds old, received at 2023-08-25T02:27:57.015228+0000: client_request(mds.1:2540 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: slow request 289.259668 seconds old, received at 2023-08-25T02:27:57.047108+0000: client_request(mds.1:2572 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:47 smithi121 ceph-mon[100099]: pgmap v191: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 162 KiB/s wr, 40 op/s 2023-08-25T02:32:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: 1031 slow requests, 5 included below; oldest blocked for > 290.685919 secs 2023-08-25T02:32:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: slow request 289.386025 seconds old, received at 2023-08-25T02:27:56.920752+0000: client_request(mds.1:2444 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: slow request 289.354085 seconds old, received at 2023-08-25T02:27:56.952691+0000: client_request(mds.1:2476 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: slow request 289.322290 seconds old, received at 2023-08-25T02:27:56.984486+0000: client_request(mds.1:2508 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: slow request 289.291548 seconds old, received at 2023-08-25T02:27:57.015228+0000: client_request(mds.1:2540 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: slow request 289.259668 seconds old, received at 2023-08-25T02:27:57.047108+0000: client_request(mds.1:2572 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:47.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:47 smithi122 ceph-mon[100457]: pgmap v191: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 162 KiB/s wr, 40 op/s 2023-08-25T02:32:48.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: 1031 slow requests, 5 included below; oldest blocked for > 290.685919 secs 2023-08-25T02:32:48.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: slow request 289.386025 seconds old, received at 2023-08-25T02:27:56.920752+0000: client_request(mds.1:2444 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:48.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: slow request 289.354085 seconds old, received at 2023-08-25T02:27:56.952691+0000: client_request(mds.1:2476 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:48.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: slow request 289.322290 seconds old, received at 2023-08-25T02:27:56.984486+0000: client_request(mds.1:2508 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:48.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: slow request 289.291548 seconds old, received at 2023-08-25T02:27:57.015228+0000: client_request(mds.1:2540 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:48.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: slow request 289.259668 seconds old, received at 2023-08-25T02:27:57.047108+0000: client_request(mds.1:2572 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:48.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:47 smithi089 ceph-mon[98732]: pgmap v191: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 162 KiB/s wr, 40 op/s 2023-08-25T02:32:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: pgmap v192: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:49.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: 1731 slow requests, 5 included below; oldest blocked for > 283.564410 secs 2023-08-25T02:32:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: slow request 283.289046 seconds old, received at 2023-08-25T02:28:06.105670+0000: peer_request:mds.1:2294 currently dispatched 2023-08-25T02:32:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: slow request 283.281815 seconds old, received at 2023-08-25T02:28:06.112901+0000: peer_request:mds.1:2326 currently dispatched 2023-08-25T02:32:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: slow request 283.274579 seconds old, received at 2023-08-25T02:28:06.120136+0000: peer_request:mds.1:2358 currently dispatched 2023-08-25T02:32:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: slow request 283.267398 seconds old, received at 2023-08-25T02:28:06.127318+0000: peer_request:mds.1:2390 currently dispatched 2023-08-25T02:32:49.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:49 smithi121 ceph-mon[100099]: slow request 283.260096 seconds old, received at 2023-08-25T02:28:06.134620+0000: peer_request:mds.1:2422 currently dispatched 2023-08-25T02:32:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: pgmap v192: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: 1731 slow requests, 5 included below; oldest blocked for > 283.564410 secs 2023-08-25T02:32:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: slow request 283.289046 seconds old, received at 2023-08-25T02:28:06.105670+0000: peer_request:mds.1:2294 currently dispatched 2023-08-25T02:32:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: slow request 283.281815 seconds old, received at 2023-08-25T02:28:06.112901+0000: peer_request:mds.1:2326 currently dispatched 2023-08-25T02:32:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: slow request 283.274579 seconds old, received at 2023-08-25T02:28:06.120136+0000: peer_request:mds.1:2358 currently dispatched 2023-08-25T02:32:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: slow request 283.267398 seconds old, received at 2023-08-25T02:28:06.127318+0000: peer_request:mds.1:2390 currently dispatched 2023-08-25T02:32:49.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:49 smithi122 ceph-mon[100457]: slow request 283.260096 seconds old, received at 2023-08-25T02:28:06.134620+0000: peer_request:mds.1:2422 currently dispatched 2023-08-25T02:32:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: pgmap v192: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: 1731 slow requests, 5 included below; oldest blocked for > 283.564410 secs 2023-08-25T02:32:50.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: slow request 283.289046 seconds old, received at 2023-08-25T02:28:06.105670+0000: peer_request:mds.1:2294 currently dispatched 2023-08-25T02:32:50.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: slow request 283.281815 seconds old, received at 2023-08-25T02:28:06.112901+0000: peer_request:mds.1:2326 currently dispatched 2023-08-25T02:32:50.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: slow request 283.274579 seconds old, received at 2023-08-25T02:28:06.120136+0000: peer_request:mds.1:2358 currently dispatched 2023-08-25T02:32:50.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: slow request 283.267398 seconds old, received at 2023-08-25T02:28:06.127318+0000: peer_request:mds.1:2390 currently dispatched 2023-08-25T02:32:50.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:49 smithi089 ceph-mon[98732]: slow request 283.260096 seconds old, received at 2023-08-25T02:28:06.134620+0000: peer_request:mds.1:2422 currently dispatched 2023-08-25T02:32:51.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:51 smithi121 ceph-mon[100099]: pgmap v193: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:51.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:51 smithi122 ceph-mon[100457]: pgmap v193: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:51 smithi089 ceph-mon[98732]: pgmap v193: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 14 op/s 2023-08-25T02:32:52.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: 1727 slow requests, 5 included below; oldest blocked for > 295.796775 secs 2023-08-25T02:32:52.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: slow request 294.340287 seconds old, received at 2023-08-25T02:27:57.077345+0000: client_request(mds.1:2604 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: slow request 294.310433 seconds old, received at 2023-08-25T02:27:57.107199+0000: client_request(mds.1:2636 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: slow request 294.278440 seconds old, received at 2023-08-25T02:27:57.139192+0000: client_request(mds.1:2668 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: slow request 294.246544 seconds old, received at 2023-08-25T02:27:57.171088+0000: client_request(mds.1:2700 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:52 smithi122 ceph-mon[100457]: slow request 294.214840 seconds old, received at 2023-08-25T02:27:57.202792+0000: client_request(mds.1:2732 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: 1727 slow requests, 5 included below; oldest blocked for > 295.796775 secs 2023-08-25T02:32:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: slow request 294.340287 seconds old, received at 2023-08-25T02:27:57.077345+0000: client_request(mds.1:2604 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: slow request 294.310433 seconds old, received at 2023-08-25T02:27:57.107199+0000: client_request(mds.1:2636 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: slow request 294.278440 seconds old, received at 2023-08-25T02:27:57.139192+0000: client_request(mds.1:2668 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: slow request 294.246544 seconds old, received at 2023-08-25T02:27:57.171088+0000: client_request(mds.1:2700 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:52 smithi089 ceph-mon[98732]: slow request 294.214840 seconds old, received at 2023-08-25T02:27:57.202792+0000: client_request(mds.1:2732 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: 1727 slow requests, 5 included below; oldest blocked for > 295.796775 secs 2023-08-25T02:32:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: slow request 294.340287 seconds old, received at 2023-08-25T02:27:57.077345+0000: client_request(mds.1:2604 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: slow request 294.310433 seconds old, received at 2023-08-25T02:27:57.107199+0000: client_request(mds.1:2636 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: slow request 294.278440 seconds old, received at 2023-08-25T02:27:57.139192+0000: client_request(mds.1:2668 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: slow request 294.246544 seconds old, received at 2023-08-25T02:27:57.171088+0000: client_request(mds.1:2700 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:52.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:52 smithi121 ceph-mon[100099]: slow request 294.214840 seconds old, received at 2023-08-25T02:27:57.202792+0000: client_request(mds.1:2732 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:53 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:53 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:53.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:53 smithi122 ceph-mon[100457]: pgmap v194: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 80 KiB/s wr, 59 op/s 2023-08-25T02:32:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:53 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:53.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:53 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:53.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:53 smithi089 ceph-mon[98732]: pgmap v194: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 80 KiB/s wr, 59 op/s 2023-08-25T02:32:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:53 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:32:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:53 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:53 smithi121 ceph-mon[100099]: pgmap v194: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 80 KiB/s wr, 59 op/s 2023-08-25T02:32:54.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:54 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:54.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:54 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:54.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:54 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:54.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:54 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:54 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:54.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:54 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:55.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: 1721 slow requests, 5 included below; oldest blocked for > 288.564539 secs 2023-08-25T02:32:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: slow request 288.252960 seconds old, received at 2023-08-25T02:28:06.141884+0000: peer_request:mds.1:2454 currently dispatched 2023-08-25T02:32:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: slow request 288.245735 seconds old, received at 2023-08-25T02:28:06.149109+0000: peer_request:mds.1:2486 currently dispatched 2023-08-25T02:32:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: slow request 288.238486 seconds old, received at 2023-08-25T02:28:06.156358+0000: peer_request:mds.1:2518 currently dispatched 2023-08-25T02:32:55.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: slow request 288.231244 seconds old, received at 2023-08-25T02:28:06.163600+0000: peer_request:mds.1:2550 currently dispatched 2023-08-25T02:32:55.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: slow request 288.223970 seconds old, received at 2023-08-25T02:28:06.170875+0000: peer_request:mds.1:2582 currently dispatched 2023-08-25T02:32:55.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:55 smithi089 ceph-mon[98732]: pgmap v195: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 51 op/s 2023-08-25T02:32:55.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: 1721 slow requests, 5 included below; oldest blocked for > 288.564539 secs 2023-08-25T02:32:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: slow request 288.252960 seconds old, received at 2023-08-25T02:28:06.141884+0000: peer_request:mds.1:2454 currently dispatched 2023-08-25T02:32:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: slow request 288.245735 seconds old, received at 2023-08-25T02:28:06.149109+0000: peer_request:mds.1:2486 currently dispatched 2023-08-25T02:32:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: slow request 288.238486 seconds old, received at 2023-08-25T02:28:06.156358+0000: peer_request:mds.1:2518 currently dispatched 2023-08-25T02:32:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: slow request 288.231244 seconds old, received at 2023-08-25T02:28:06.163600+0000: peer_request:mds.1:2550 currently dispatched 2023-08-25T02:32:55.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: slow request 288.223970 seconds old, received at 2023-08-25T02:28:06.170875+0000: peer_request:mds.1:2582 currently dispatched 2023-08-25T02:32:55.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:55 smithi121 ceph-mon[100099]: pgmap v195: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 51 op/s 2023-08-25T02:32:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: 1721 slow requests, 5 included below; oldest blocked for > 288.564539 secs 2023-08-25T02:32:55.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: slow request 288.252960 seconds old, received at 2023-08-25T02:28:06.141884+0000: peer_request:mds.1:2454 currently dispatched 2023-08-25T02:32:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: slow request 288.245735 seconds old, received at 2023-08-25T02:28:06.149109+0000: peer_request:mds.1:2486 currently dispatched 2023-08-25T02:32:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: slow request 288.238486 seconds old, received at 2023-08-25T02:28:06.156358+0000: peer_request:mds.1:2518 currently dispatched 2023-08-25T02:32:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: slow request 288.231244 seconds old, received at 2023-08-25T02:28:06.163600+0000: peer_request:mds.1:2550 currently dispatched 2023-08-25T02:32:55.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: slow request 288.223970 seconds old, received at 2023-08-25T02:28:06.170875+0000: peer_request:mds.1:2582 currently dispatched 2023-08-25T02:32:55.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:55 smithi122 ceph-mon[100457]: pgmap v195: 97 pgs: 97 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 51 op/s 2023-08-25T02:32:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: pgmap v196: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 666 op/s 2023-08-25T02:32:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: 1446 slow requests, 5 included below; oldest blocked for > 300.880660 secs 2023-08-25T02:32:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: slow request 299.266902 seconds old, received at 2023-08-25T02:27:57.234615+0000: client_request(mds.1:2764 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: slow request 299.235945 seconds old, received at 2023-08-25T02:27:57.265572+0000: client_request(mds.1:2796 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: slow request 299.207444 seconds old, received at 2023-08-25T02:27:57.294074+0000: client_request(mds.1:2828 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: slow request 299.178709 seconds old, received at 2023-08-25T02:27:57.322808+0000: client_request(mds.1:2860 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:57 smithi089 ceph-mon[98732]: slow request 299.149563 seconds old, received at 2023-08-25T02:27:57.351954+0000: client_request(mds.1:2892 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: pgmap v196: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 666 op/s 2023-08-25T02:32:57.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: 1446 slow requests, 5 included below; oldest blocked for > 300.880660 secs 2023-08-25T02:32:57.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: slow request 299.266902 seconds old, received at 2023-08-25T02:27:57.234615+0000: client_request(mds.1:2764 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: slow request 299.235945 seconds old, received at 2023-08-25T02:27:57.265572+0000: client_request(mds.1:2796 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: slow request 299.207444 seconds old, received at 2023-08-25T02:27:57.294074+0000: client_request(mds.1:2828 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: slow request 299.178709 seconds old, received at 2023-08-25T02:27:57.322808+0000: client_request(mds.1:2860 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:57 smithi121 ceph-mon[100099]: slow request 299.149563 seconds old, received at 2023-08-25T02:27:57.351954+0000: client_request(mds.1:2892 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: pgmap v196: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 666 op/s 2023-08-25T02:32:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: 1446 slow requests, 5 included below; oldest blocked for > 300.880660 secs 2023-08-25T02:32:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: slow request 299.266902 seconds old, received at 2023-08-25T02:27:57.234615+0000: client_request(mds.1:2764 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: slow request 299.235945 seconds old, received at 2023-08-25T02:27:57.265572+0000: client_request(mds.1:2796 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: slow request 299.207444 seconds old, received at 2023-08-25T02:27:57.294074+0000: client_request(mds.1:2828 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: slow request 299.178709 seconds old, received at 2023-08-25T02:27:57.322808+0000: client_request(mds.1:2860 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:57.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:57 smithi122 ceph-mon[100457]: slow request 299.149563 seconds old, received at 2023-08-25T02:27:57.351954+0000: client_request(mds.1:2892 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:32:58.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:58 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:58.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:58 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:58.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:58 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:58.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:58 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:58.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:58 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:32:58.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:58 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:32:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: pgmap v197: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:32:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: 1711 slow requests, 5 included below; oldest blocked for > 293.564647 secs 2023-08-25T02:32:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: slow request 293.217876 seconds old, received at 2023-08-25T02:28:06.177077+0000: peer_request:mds.1:2614 currently dispatched 2023-08-25T02:32:59.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: slow request 293.211605 seconds old, received at 2023-08-25T02:28:06.183347+0000: peer_request:mds.1:2646 currently dispatched 2023-08-25T02:32:59.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: slow request 293.205253 seconds old, received at 2023-08-25T02:28:06.189699+0000: peer_request:mds.1:2678 currently dispatched 2023-08-25T02:32:59.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: slow request 293.198958 seconds old, received at 2023-08-25T02:28:06.195995+0000: peer_request:mds.1:2710 currently dispatched 2023-08-25T02:32:59.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:32:59 smithi089 ceph-mon[98732]: slow request 293.192618 seconds old, received at 2023-08-25T02:28:06.202335+0000: peer_request:mds.1:2742 currently dispatched 2023-08-25T02:32:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: pgmap v197: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:32:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: 1711 slow requests, 5 included below; oldest blocked for > 293.564647 secs 2023-08-25T02:32:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: slow request 293.217876 seconds old, received at 2023-08-25T02:28:06.177077+0000: peer_request:mds.1:2614 currently dispatched 2023-08-25T02:32:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: slow request 293.211605 seconds old, received at 2023-08-25T02:28:06.183347+0000: peer_request:mds.1:2646 currently dispatched 2023-08-25T02:32:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: slow request 293.205253 seconds old, received at 2023-08-25T02:28:06.189699+0000: peer_request:mds.1:2678 currently dispatched 2023-08-25T02:32:59.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: slow request 293.198958 seconds old, received at 2023-08-25T02:28:06.195995+0000: peer_request:mds.1:2710 currently dispatched 2023-08-25T02:32:59.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:32:59 smithi121 ceph-mon[100099]: slow request 293.192618 seconds old, received at 2023-08-25T02:28:06.202335+0000: peer_request:mds.1:2742 currently dispatched 2023-08-25T02:32:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: pgmap v197: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:32:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: 1711 slow requests, 5 included below; oldest blocked for > 293.564647 secs 2023-08-25T02:32:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: slow request 293.217876 seconds old, received at 2023-08-25T02:28:06.177077+0000: peer_request:mds.1:2614 currently dispatched 2023-08-25T02:32:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: slow request 293.211605 seconds old, received at 2023-08-25T02:28:06.183347+0000: peer_request:mds.1:2646 currently dispatched 2023-08-25T02:32:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: slow request 293.205253 seconds old, received at 2023-08-25T02:28:06.189699+0000: peer_request:mds.1:2678 currently dispatched 2023-08-25T02:32:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: slow request 293.198958 seconds old, received at 2023-08-25T02:28:06.195995+0000: peer_request:mds.1:2710 currently dispatched 2023-08-25T02:32:59.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:32:59 smithi122 ceph-mon[100457]: slow request 293.192618 seconds old, received at 2023-08-25T02:28:06.202335+0000: peer_request:mds.1:2742 currently dispatched 2023-08-25T02:33:01.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:01 smithi089 ceph-mon[98732]: pgmap v198: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:33:01.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:01 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:01.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:01 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:01.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:01 smithi121 ceph-mon[100099]: pgmap v198: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:33:01.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:01 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:01.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:01 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:01 smithi122 ceph-mon[100457]: pgmap v198: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 659 op/s 2023-08-25T02:33:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:01 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:01.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:01 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:02.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: 216 slow requests, 5 included below; oldest blocked for > 305.956438 secs 2023-08-25T02:33:02.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: slow request 304.195586 seconds old, received at 2023-08-25T02:27:57.381709+0000: client_request(mds.1:2924 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: slow request 304.163330 seconds old, received at 2023-08-25T02:27:57.413965+0000: client_request(mds.1:2956 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: slow request 304.129871 seconds old, received at 2023-08-25T02:27:57.447424+0000: client_request(mds.1:2988 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: slow request 304.099674 seconds old, received at 2023-08-25T02:27:57.477621+0000: client_request(mds.1:3020 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:02 smithi089 ceph-mon[98732]: slow request 304.069807 seconds old, received at 2023-08-25T02:27:57.507488+0000: client_request(mds.1:3052 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:02.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:02.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: 216 slow requests, 5 included below; oldest blocked for > 305.956438 secs 2023-08-25T02:33:02.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: slow request 304.195586 seconds old, received at 2023-08-25T02:27:57.381709+0000: client_request(mds.1:2924 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: slow request 304.163330 seconds old, received at 2023-08-25T02:27:57.413965+0000: client_request(mds.1:2956 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: slow request 304.129871 seconds old, received at 2023-08-25T02:27:57.447424+0000: client_request(mds.1:2988 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: slow request 304.099674 seconds old, received at 2023-08-25T02:27:57.477621+0000: client_request(mds.1:3020 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:02 smithi121 ceph-mon[100099]: slow request 304.069807 seconds old, received at 2023-08-25T02:27:57.507488+0000: client_request(mds.1:3052 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:02.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:02.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: 216 slow requests, 5 included below; oldest blocked for > 305.956438 secs 2023-08-25T02:33:02.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: slow request 304.195586 seconds old, received at 2023-08-25T02:27:57.381709+0000: client_request(mds.1:2924 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: slow request 304.163330 seconds old, received at 2023-08-25T02:27:57.413965+0000: client_request(mds.1:2956 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: slow request 304.129871 seconds old, received at 2023-08-25T02:27:57.447424+0000: client_request(mds.1:2988 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: slow request 304.099674 seconds old, received at 2023-08-25T02:27:57.477621+0000: client_request(mds.1:3020 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:02.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:02 smithi122 ceph-mon[100457]: slow request 304.069807 seconds old, received at 2023-08-25T02:27:57.507488+0000: client_request(mds.1:3052 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:03 smithi089 ceph-mon[98732]: pgmap v199: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 676 op/s 2023-08-25T02:33:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:03 smithi121 ceph-mon[100099]: pgmap v199: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 676 op/s 2023-08-25T02:33:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:03 smithi122 ceph-mon[100457]: pgmap v199: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 7.9 MiB/s wr, 676 op/s 2023-08-25T02:33:04.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: 905 slow requests, 5 included below; oldest blocked for > 298.564747 secs 2023-08-25T02:33:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: slow request 298.186345 seconds old, received at 2023-08-25T02:28:06.208706+0000: peer_request:mds.1:2774 currently dispatched 2023-08-25T02:33:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: slow request 298.179293 seconds old, received at 2023-08-25T02:28:06.215759+0000: peer_request:mds.1:2806 currently dispatched 2023-08-25T02:33:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: slow request 298.172510 seconds old, received at 2023-08-25T02:28:06.222542+0000: peer_request:mds.1:2838 currently dispatched 2023-08-25T02:33:04.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: slow request 298.166292 seconds old, received at 2023-08-25T02:28:06.228760+0000: peer_request:mds.1:2870 currently dispatched 2023-08-25T02:33:04.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:04 smithi089 ceph-mon[98732]: slow request 298.159894 seconds old, received at 2023-08-25T02:28:06.235157+0000: peer_request:mds.1:2902 currently dispatched 2023-08-25T02:33:04.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: 905 slow requests, 5 included below; oldest blocked for > 298.564747 secs 2023-08-25T02:33:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: slow request 298.186345 seconds old, received at 2023-08-25T02:28:06.208706+0000: peer_request:mds.1:2774 currently dispatched 2023-08-25T02:33:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: slow request 298.179293 seconds old, received at 2023-08-25T02:28:06.215759+0000: peer_request:mds.1:2806 currently dispatched 2023-08-25T02:33:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: slow request 298.172510 seconds old, received at 2023-08-25T02:28:06.222542+0000: peer_request:mds.1:2838 currently dispatched 2023-08-25T02:33:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: slow request 298.166292 seconds old, received at 2023-08-25T02:28:06.228760+0000: peer_request:mds.1:2870 currently dispatched 2023-08-25T02:33:04.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:04 smithi121 ceph-mon[100099]: slow request 298.159894 seconds old, received at 2023-08-25T02:28:06.235157+0000: peer_request:mds.1:2902 currently dispatched 2023-08-25T02:33:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: 905 slow requests, 5 included below; oldest blocked for > 298.564747 secs 2023-08-25T02:33:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: slow request 298.186345 seconds old, received at 2023-08-25T02:28:06.208706+0000: peer_request:mds.1:2774 currently dispatched 2023-08-25T02:33:04.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: slow request 298.179293 seconds old, received at 2023-08-25T02:28:06.215759+0000: peer_request:mds.1:2806 currently dispatched 2023-08-25T02:33:04.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: slow request 298.172510 seconds old, received at 2023-08-25T02:28:06.222542+0000: peer_request:mds.1:2838 currently dispatched 2023-08-25T02:33:04.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: slow request 298.166292 seconds old, received at 2023-08-25T02:28:06.228760+0000: peer_request:mds.1:2870 currently dispatched 2023-08-25T02:33:04.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:04 smithi122 ceph-mon[100457]: slow request 298.159894 seconds old, received at 2023-08-25T02:28:06.235157+0000: peer_request:mds.1:2902 currently dispatched 2023-08-25T02:33:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:05 smithi089 ceph-mon[98732]: pgmap v200: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 631 op/s 2023-08-25T02:33:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:05 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:05.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:05 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:05 smithi121 ceph-mon[100099]: pgmap v200: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 631 op/s 2023-08-25T02:33:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:05 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:05.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:05 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:05 smithi122 ceph-mon[100457]: pgmap v200: 97 pgs: 97 active+clean; 8.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 631 op/s 2023-08-25T02:33:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:05 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:05 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:06.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:06 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:06.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:06 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:06.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:06 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:06 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:06.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:06 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:06 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: pgmap v201: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 22 MiB/s wr, 797 op/s 2023-08-25T02:33:07.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: 526 slow requests, 5 included below; oldest blocked for > 311.161031 secs 2023-08-25T02:33:07.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: slow request 309.245608 seconds old, received at 2023-08-25T02:27:57.536280+0000: client_request(mds.1:3084 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: slow request 309.216546 seconds old, received at 2023-08-25T02:27:57.565342+0000: client_request(mds.1:3116 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: slow request 309.187656 seconds old, received at 2023-08-25T02:27:57.594232+0000: client_request(mds.1:3148 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: slow request 309.154725 seconds old, received at 2023-08-25T02:27:57.627163+0000: client_request(mds.1:3180 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:07 smithi089 ceph-mon[98732]: slow request 309.121497 seconds old, received at 2023-08-25T02:27:57.660391+0000: client_request(mds.1:3212 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: pgmap v201: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 22 MiB/s wr, 797 op/s 2023-08-25T02:33:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: 526 slow requests, 5 included below; oldest blocked for > 311.161031 secs 2023-08-25T02:33:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: slow request 309.245608 seconds old, received at 2023-08-25T02:27:57.536280+0000: client_request(mds.1:3084 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: slow request 309.216546 seconds old, received at 2023-08-25T02:27:57.565342+0000: client_request(mds.1:3116 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: slow request 309.187656 seconds old, received at 2023-08-25T02:27:57.594232+0000: client_request(mds.1:3148 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: slow request 309.154725 seconds old, received at 2023-08-25T02:27:57.627163+0000: client_request(mds.1:3180 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:07 smithi121 ceph-mon[100099]: slow request 309.121497 seconds old, received at 2023-08-25T02:27:57.660391+0000: client_request(mds.1:3212 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: pgmap v201: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 22 MiB/s wr, 797 op/s 2023-08-25T02:33:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: 526 slow requests, 5 included below; oldest blocked for > 311.161031 secs 2023-08-25T02:33:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: slow request 309.245608 seconds old, received at 2023-08-25T02:27:57.536280+0000: client_request(mds.1:3084 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: slow request 309.216546 seconds old, received at 2023-08-25T02:27:57.565342+0000: client_request(mds.1:3116 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: slow request 309.187656 seconds old, received at 2023-08-25T02:27:57.594232+0000: client_request(mds.1:3148 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: slow request 309.154725 seconds old, received at 2023-08-25T02:27:57.627163+0000: client_request(mds.1:3180 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:07.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:07 smithi122 ceph-mon[100457]: slow request 309.121497 seconds old, received at 2023-08-25T02:27:57.660391+0000: client_request(mds.1:3212 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: pgmap v202: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 15 MiB/s wr, 182 op/s 2023-08-25T02:33:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:09.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: 1010 slow requests, 5 included below; oldest blocked for > 303.564827 secs 2023-08-25T02:33:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: slow request 303.153725 seconds old, received at 2023-08-25T02:28:06.241408+0000: peer_request:mds.1:2934 currently dispatched 2023-08-25T02:33:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: slow request 303.147371 seconds old, received at 2023-08-25T02:28:06.247762+0000: peer_request:mds.1:2966 currently dispatched 2023-08-25T02:33:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: slow request 303.140140 seconds old, received at 2023-08-25T02:28:06.254993+0000: peer_request:mds.1:2998 currently dispatched 2023-08-25T02:33:09.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: slow request 303.132941 seconds old, received at 2023-08-25T02:28:06.262192+0000: peer_request:mds.1:3030 currently dispatched 2023-08-25T02:33:09.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:09 smithi121 ceph-mon[100099]: slow request 303.125636 seconds old, received at 2023-08-25T02:28:06.269497+0000: peer_request:mds.1:3062 currently dispatched 2023-08-25T02:33:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: pgmap v202: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 15 MiB/s wr, 182 op/s 2023-08-25T02:33:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: 1010 slow requests, 5 included below; oldest blocked for > 303.564827 secs 2023-08-25T02:33:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: slow request 303.153725 seconds old, received at 2023-08-25T02:28:06.241408+0000: peer_request:mds.1:2934 currently dispatched 2023-08-25T02:33:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: slow request 303.147371 seconds old, received at 2023-08-25T02:28:06.247762+0000: peer_request:mds.1:2966 currently dispatched 2023-08-25T02:33:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: slow request 303.140140 seconds old, received at 2023-08-25T02:28:06.254993+0000: peer_request:mds.1:2998 currently dispatched 2023-08-25T02:33:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: slow request 303.132941 seconds old, received at 2023-08-25T02:28:06.262192+0000: peer_request:mds.1:3030 currently dispatched 2023-08-25T02:33:09.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:09 smithi122 ceph-mon[100457]: slow request 303.125636 seconds old, received at 2023-08-25T02:28:06.269497+0000: peer_request:mds.1:3062 currently dispatched 2023-08-25T02:33:10.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: pgmap v202: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 15 MiB/s wr, 182 op/s 2023-08-25T02:33:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: 1010 slow requests, 5 included below; oldest blocked for > 303.564827 secs 2023-08-25T02:33:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: slow request 303.153725 seconds old, received at 2023-08-25T02:28:06.241408+0000: peer_request:mds.1:2934 currently dispatched 2023-08-25T02:33:10.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: slow request 303.147371 seconds old, received at 2023-08-25T02:28:06.247762+0000: peer_request:mds.1:2966 currently dispatched 2023-08-25T02:33:10.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: slow request 303.140140 seconds old, received at 2023-08-25T02:28:06.254993+0000: peer_request:mds.1:2998 currently dispatched 2023-08-25T02:33:10.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: slow request 303.132941 seconds old, received at 2023-08-25T02:28:06.262192+0000: peer_request:mds.1:3030 currently dispatched 2023-08-25T02:33:10.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:09 smithi089 ceph-mon[98732]: slow request 303.125636 seconds old, received at 2023-08-25T02:28:06.269497+0000: peer_request:mds.1:3062 currently dispatched 2023-08-25T02:33:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:10 smithi121 ceph-mon[100099]: osdmap e84: 12 total, 12 up, 12 in 2023-08-25T02:33:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:10 smithi122 ceph-mon[100457]: osdmap e84: 12 total, 12 up, 12 in 2023-08-25T02:33:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:10 smithi089 ceph-mon[98732]: osdmap e84: 12 total, 12 up, 12 in 2023-08-25T02:33:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:11 smithi121 ceph-mon[100099]: pgmap v204: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 219 op/s 2023-08-25T02:33:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:11 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:11 smithi122 ceph-mon[100457]: pgmap v204: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 219 op/s 2023-08-25T02:33:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:11 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:11 smithi089 ceph-mon[98732]: pgmap v204: 97 pgs: 97 active+clean; 8.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 219 op/s 2023-08-25T02:33:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:11 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: 1688 slow requests, 5 included below; oldest blocked for > 316.322821 secs 2023-08-25T02:33:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: slow request 314.254157 seconds old, received at 2023-08-25T02:27:57.689521+0000: client_request(mds.1:3244 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: slow request 314.225315 seconds old, received at 2023-08-25T02:27:57.718363+0000: client_request(mds.1:3276 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: slow request 315.832050 seconds old, received at 2023-08-25T02:27:56.111628+0000: client_request(mds.1:1613 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: slow request 315.798511 seconds old, received at 2023-08-25T02:27:56.145168+0000: client_request(mds.1:1645 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: slow request 315.764827 seconds old, received at 2023-08-25T02:27:56.178851+0000: client_request(mds.1:1677 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:13 smithi121 ceph-mon[100099]: pgmap v205: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: 1688 slow requests, 5 included below; oldest blocked for > 316.322821 secs 2023-08-25T02:33:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: slow request 314.254157 seconds old, received at 2023-08-25T02:27:57.689521+0000: client_request(mds.1:3244 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: slow request 314.225315 seconds old, received at 2023-08-25T02:27:57.718363+0000: client_request(mds.1:3276 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: slow request 315.832050 seconds old, received at 2023-08-25T02:27:56.111628+0000: client_request(mds.1:1613 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: slow request 315.798511 seconds old, received at 2023-08-25T02:27:56.145168+0000: client_request(mds.1:1645 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: slow request 315.764827 seconds old, received at 2023-08-25T02:27:56.178851+0000: client_request(mds.1:1677 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:13.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:13 smithi122 ceph-mon[100457]: pgmap v205: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: 1688 slow requests, 5 included below; oldest blocked for > 316.322821 secs 2023-08-25T02:33:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: slow request 314.254157 seconds old, received at 2023-08-25T02:27:57.689521+0000: client_request(mds.1:3244 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: slow request 314.225315 seconds old, received at 2023-08-25T02:27:57.718363+0000: client_request(mds.1:3276 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: slow request 315.832050 seconds old, received at 2023-08-25T02:27:56.111628+0000: client_request(mds.1:1613 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: slow request 315.798511 seconds old, received at 2023-08-25T02:27:56.145168+0000: client_request(mds.1:1645 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:14.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: slow request 315.764827 seconds old, received at 2023-08-25T02:27:56.178851+0000: client_request(mds.1:1677 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:14.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:13 smithi089 ceph-mon[98732]: pgmap v205: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: 1684 slow requests, 5 included below; oldest blocked for > 308.564968 secs 2023-08-25T02:33:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: slow request 308.118517 seconds old, received at 2023-08-25T02:28:06.276756+0000: peer_request:mds.1:3094 currently dispatched 2023-08-25T02:33:14.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: slow request 308.111164 seconds old, received at 2023-08-25T02:28:06.284109+0000: peer_request:mds.1:3126 currently dispatched 2023-08-25T02:33:14.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: slow request 308.104319 seconds old, received at 2023-08-25T02:28:06.290955+0000: peer_request:mds.1:3158 currently dispatched 2023-08-25T02:33:14.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: slow request 308.098397 seconds old, received at 2023-08-25T02:28:06.296877+0000: peer_request:mds.1:3190 currently dispatched 2023-08-25T02:33:14.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:14 smithi121 ceph-mon[100099]: slow request 308.091916 seconds old, received at 2023-08-25T02:28:06.303358+0000: peer_request:mds.1:3222 currently dispatched 2023-08-25T02:33:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: 1684 slow requests, 5 included below; oldest blocked for > 308.564968 secs 2023-08-25T02:33:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: slow request 308.118517 seconds old, received at 2023-08-25T02:28:06.276756+0000: peer_request:mds.1:3094 currently dispatched 2023-08-25T02:33:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: slow request 308.111164 seconds old, received at 2023-08-25T02:28:06.284109+0000: peer_request:mds.1:3126 currently dispatched 2023-08-25T02:33:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: slow request 308.104319 seconds old, received at 2023-08-25T02:28:06.290955+0000: peer_request:mds.1:3158 currently dispatched 2023-08-25T02:33:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: slow request 308.098397 seconds old, received at 2023-08-25T02:28:06.296877+0000: peer_request:mds.1:3190 currently dispatched 2023-08-25T02:33:14.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:14 smithi122 ceph-mon[100457]: slow request 308.091916 seconds old, received at 2023-08-25T02:28:06.303358+0000: peer_request:mds.1:3222 currently dispatched 2023-08-25T02:33:15.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:15.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:15.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: 1684 slow requests, 5 included below; oldest blocked for > 308.564968 secs 2023-08-25T02:33:15.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: slow request 308.118517 seconds old, received at 2023-08-25T02:28:06.276756+0000: peer_request:mds.1:3094 currently dispatched 2023-08-25T02:33:15.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: slow request 308.111164 seconds old, received at 2023-08-25T02:28:06.284109+0000: peer_request:mds.1:3126 currently dispatched 2023-08-25T02:33:15.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: slow request 308.104319 seconds old, received at 2023-08-25T02:28:06.290955+0000: peer_request:mds.1:3158 currently dispatched 2023-08-25T02:33:15.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: slow request 308.098397 seconds old, received at 2023-08-25T02:28:06.296877+0000: peer_request:mds.1:3190 currently dispatched 2023-08-25T02:33:15.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:14 smithi089 ceph-mon[98732]: slow request 308.091916 seconds old, received at 2023-08-25T02:28:06.303358+0000: peer_request:mds.1:3222 currently dispatched 2023-08-25T02:33:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:15 smithi121 ceph-mon[100099]: pgmap v206: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:15 smithi121 ceph-mon[100099]: osdmap e85: 12 total, 12 up, 12 in 2023-08-25T02:33:15.977 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:15 smithi122 ceph-mon[100457]: pgmap v206: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:15 smithi122 ceph-mon[100457]: osdmap e85: 12 total, 12 up, 12 in 2023-08-25T02:33:16.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:15 smithi089 ceph-mon[98732]: pgmap v206: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 204 op/s 2023-08-25T02:33:16.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:15 smithi089 ceph-mon[98732]: osdmap e85: 12 total, 12 up, 12 in 2023-08-25T02:33:17.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:17 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:17.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:17 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:17.479 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:17 smithi122 ceph-mon[100457]: pgmap v208: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-08-25T02:33:17.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:17 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:17.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:17 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:17.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:17 smithi089 ceph-mon[98732]: pgmap v208: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-08-25T02:33:17.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:17 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:17.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:17 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:17.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:17 smithi121 ceph-mon[100099]: pgmap v208: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-08-25T02:33:18.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: 1679 slow requests, 5 included below; oldest blocked for > 321.349239 secs 2023-08-25T02:33:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: slow request 320.757845 seconds old, received at 2023-08-25T02:27:56.212251+0000: client_request(mds.1:1709 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: slow request 320.724310 seconds old, received at 2023-08-25T02:27:56.245785+0000: client_request(mds.1:1741 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: slow request 320.691526 seconds old, received at 2023-08-25T02:27:56.278570+0000: client_request(mds.1:1773 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: slow request 320.658712 seconds old, received at 2023-08-25T02:27:56.311384+0000: client_request(mds.1:1805 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: slow request 320.625388 seconds old, received at 2023-08-25T02:27:56.344708+0000: client_request(mds.1:1837 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:18.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:18 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:18.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: 1679 slow requests, 5 included below; oldest blocked for > 321.349239 secs 2023-08-25T02:33:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: slow request 320.757845 seconds old, received at 2023-08-25T02:27:56.212251+0000: client_request(mds.1:1709 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: slow request 320.724310 seconds old, received at 2023-08-25T02:27:56.245785+0000: client_request(mds.1:1741 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: slow request 320.691526 seconds old, received at 2023-08-25T02:27:56.278570+0000: client_request(mds.1:1773 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: slow request 320.658712 seconds old, received at 2023-08-25T02:27:56.311384+0000: client_request(mds.1:1805 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: slow request 320.625388 seconds old, received at 2023-08-25T02:27:56.344708+0000: client_request(mds.1:1837 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:18.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:18 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: 1679 slow requests, 5 included below; oldest blocked for > 321.349239 secs 2023-08-25T02:33:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: slow request 320.757845 seconds old, received at 2023-08-25T02:27:56.212251+0000: client_request(mds.1:1709 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: slow request 320.724310 seconds old, received at 2023-08-25T02:27:56.245785+0000: client_request(mds.1:1741 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: slow request 320.691526 seconds old, received at 2023-08-25T02:27:56.278570+0000: client_request(mds.1:1773 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: slow request 320.658712 seconds old, received at 2023-08-25T02:27:56.311384+0000: client_request(mds.1:1805 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: slow request 320.625388 seconds old, received at 2023-08-25T02:27:56.344708+0000: client_request(mds.1:1837 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:18.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:18.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:18 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:19.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:19 smithi089 ceph-mon[98732]: pgmap v209: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:19.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:19 smithi121 ceph-mon[100099]: pgmap v209: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:19.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:19 smithi122 ceph-mon[100457]: pgmap v209: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:20.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: 1673 slow requests, 5 included below; oldest blocked for > 313.565114 secs 2023-08-25T02:33:20.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: slow request 313.084764 seconds old, received at 2023-08-25T02:28:06.310656+0000: peer_request:mds.1:3254 currently dispatched 2023-08-25T02:33:20.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: slow request 313.445749 seconds old, received at 2023-08-25T02:28:05.949671+0000: peer_request:mds.1:1623 currently dispatched 2023-08-25T02:33:20.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: slow request 313.437904 seconds old, received at 2023-08-25T02:28:05.957516+0000: peer_request:mds.1:1655 currently dispatched 2023-08-25T02:33:20.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: slow request 313.430100 seconds old, received at 2023-08-25T02:28:05.965320+0000: peer_request:mds.1:1687 currently dispatched 2023-08-25T02:33:20.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:20 smithi089 ceph-mon[98732]: slow request 313.422314 seconds old, received at 2023-08-25T02:28:05.973105+0000: peer_request:mds.1:1719 currently dispatched 2023-08-25T02:33:20.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: 1673 slow requests, 5 included below; oldest blocked for > 313.565114 secs 2023-08-25T02:33:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: slow request 313.084764 seconds old, received at 2023-08-25T02:28:06.310656+0000: peer_request:mds.1:3254 currently dispatched 2023-08-25T02:33:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: slow request 313.445749 seconds old, received at 2023-08-25T02:28:05.949671+0000: peer_request:mds.1:1623 currently dispatched 2023-08-25T02:33:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: slow request 313.437904 seconds old, received at 2023-08-25T02:28:05.957516+0000: peer_request:mds.1:1655 currently dispatched 2023-08-25T02:33:20.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: slow request 313.430100 seconds old, received at 2023-08-25T02:28:05.965320+0000: peer_request:mds.1:1687 currently dispatched 2023-08-25T02:33:20.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:20 smithi121 ceph-mon[100099]: slow request 313.422314 seconds old, received at 2023-08-25T02:28:05.973105+0000: peer_request:mds.1:1719 currently dispatched 2023-08-25T02:33:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: 1673 slow requests, 5 included below; oldest blocked for > 313.565114 secs 2023-08-25T02:33:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: slow request 313.084764 seconds old, received at 2023-08-25T02:28:06.310656+0000: peer_request:mds.1:3254 currently dispatched 2023-08-25T02:33:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: slow request 313.445749 seconds old, received at 2023-08-25T02:28:05.949671+0000: peer_request:mds.1:1623 currently dispatched 2023-08-25T02:33:20.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: slow request 313.437904 seconds old, received at 2023-08-25T02:28:05.957516+0000: peer_request:mds.1:1655 currently dispatched 2023-08-25T02:33:20.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: slow request 313.430100 seconds old, received at 2023-08-25T02:28:05.965320+0000: peer_request:mds.1:1687 currently dispatched 2023-08-25T02:33:20.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:20 smithi122 ceph-mon[100457]: slow request 313.422314 seconds old, received at 2023-08-25T02:28:05.973105+0000: peer_request:mds.1:1719 currently dispatched 2023-08-25T02:33:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:21 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:21 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:21.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:21 smithi089 ceph-mon[98732]: pgmap v210: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 68 KiB/s wr, 15 op/s 2023-08-25T02:33:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:21 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:21 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:21.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:21 smithi121 ceph-mon[100099]: pgmap v210: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 68 KiB/s wr, 15 op/s 2023-08-25T02:33:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:21 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:21 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:21 smithi122 ceph-mon[100457]: pgmap v210: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 68 KiB/s wr, 15 op/s 2023-08-25T02:33:22.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:22 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:22.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:22 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:22.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:22 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:22.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:22 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:22 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:22.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:22 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:23.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: 1668 slow requests, 5 included below; oldest blocked for > 326.460183 secs 2023-08-25T02:33:23.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: slow request 325.703103 seconds old, received at 2023-08-25T02:27:56.377937+0000: client_request(mds.1:1869 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: slow request 325.672654 seconds old, received at 2023-08-25T02:27:56.408386+0000: client_request(mds.1:1901 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: slow request 325.641087 seconds old, received at 2023-08-25T02:27:56.439953+0000: client_request(mds.1:1933 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: slow request 325.609646 seconds old, received at 2023-08-25T02:27:56.471395+0000: client_request(mds.1:1965 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: slow request 325.578910 seconds old, received at 2023-08-25T02:27:56.502130+0000: client_request(mds.1:1997 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:23 smithi089 ceph-mon[98732]: pgmap v211: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:23.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: 1668 slow requests, 5 included below; oldest blocked for > 326.460183 secs 2023-08-25T02:33:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: slow request 325.703103 seconds old, received at 2023-08-25T02:27:56.377937+0000: client_request(mds.1:1869 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: slow request 325.672654 seconds old, received at 2023-08-25T02:27:56.408386+0000: client_request(mds.1:1901 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: slow request 325.641087 seconds old, received at 2023-08-25T02:27:56.439953+0000: client_request(mds.1:1933 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: slow request 325.609646 seconds old, received at 2023-08-25T02:27:56.471395+0000: client_request(mds.1:1965 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: slow request 325.578910 seconds old, received at 2023-08-25T02:27:56.502130+0000: client_request(mds.1:1997 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:23 smithi121 ceph-mon[100099]: pgmap v211: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: 1668 slow requests, 5 included below; oldest blocked for > 326.460183 secs 2023-08-25T02:33:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: slow request 325.703103 seconds old, received at 2023-08-25T02:27:56.377937+0000: client_request(mds.1:1869 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: slow request 325.672654 seconds old, received at 2023-08-25T02:27:56.408386+0000: client_request(mds.1:1901 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: slow request 325.641087 seconds old, received at 2023-08-25T02:27:56.439953+0000: client_request(mds.1:1933 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: slow request 325.609646 seconds old, received at 2023-08-25T02:27:56.471395+0000: client_request(mds.1:1965 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: slow request 325.578910 seconds old, received at 2023-08-25T02:27:56.502130+0000: client_request(mds.1:1997 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:23 smithi122 ceph-mon[100457]: pgmap v211: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:25.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:25.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: 1664 slow requests, 5 included below; oldest blocked for > 318.565212 secs 2023-08-25T02:33:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: slow request 318.414788 seconds old, received at 2023-08-25T02:28:05.980730+0000: peer_request:mds.1:1751 currently dispatched 2023-08-25T02:33:25.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: slow request 318.407608 seconds old, received at 2023-08-25T02:28:05.987910+0000: peer_request:mds.1:1783 currently dispatched 2023-08-25T02:33:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: slow request 318.400447 seconds old, received at 2023-08-25T02:28:05.995070+0000: peer_request:mds.1:1815 currently dispatched 2023-08-25T02:33:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: slow request 318.392861 seconds old, received at 2023-08-25T02:28:06.002657+0000: peer_request:mds.1:1847 currently dispatched 2023-08-25T02:33:25.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: slow request 318.385052 seconds old, received at 2023-08-25T02:28:06.010465+0000: peer_request:mds.1:1879 currently dispatched 2023-08-25T02:33:25.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:25 smithi089 ceph-mon[98732]: pgmap v212: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:25.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:25.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:25.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: 1664 slow requests, 5 included below; oldest blocked for > 318.565212 secs 2023-08-25T02:33:25.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: slow request 318.414788 seconds old, received at 2023-08-25T02:28:05.980730+0000: peer_request:mds.1:1751 currently dispatched 2023-08-25T02:33:25.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: slow request 318.407608 seconds old, received at 2023-08-25T02:28:05.987910+0000: peer_request:mds.1:1783 currently dispatched 2023-08-25T02:33:25.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: slow request 318.400447 seconds old, received at 2023-08-25T02:28:05.995070+0000: peer_request:mds.1:1815 currently dispatched 2023-08-25T02:33:25.721 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: slow request 318.392861 seconds old, received at 2023-08-25T02:28:06.002657+0000: peer_request:mds.1:1847 currently dispatched 2023-08-25T02:33:25.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: slow request 318.385052 seconds old, received at 2023-08-25T02:28:06.010465+0000: peer_request:mds.1:1879 currently dispatched 2023-08-25T02:33:25.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:25 smithi121 ceph-mon[100099]: pgmap v212: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:25.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:25.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: 1664 slow requests, 5 included below; oldest blocked for > 318.565212 secs 2023-08-25T02:33:25.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: slow request 318.414788 seconds old, received at 2023-08-25T02:28:05.980730+0000: peer_request:mds.1:1751 currently dispatched 2023-08-25T02:33:25.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: slow request 318.407608 seconds old, received at 2023-08-25T02:28:05.987910+0000: peer_request:mds.1:1783 currently dispatched 2023-08-25T02:33:25.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: slow request 318.400447 seconds old, received at 2023-08-25T02:28:05.995070+0000: peer_request:mds.1:1815 currently dispatched 2023-08-25T02:33:25.730 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: slow request 318.392861 seconds old, received at 2023-08-25T02:28:06.002657+0000: peer_request:mds.1:1847 currently dispatched 2023-08-25T02:33:25.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: slow request 318.385052 seconds old, received at 2023-08-25T02:28:06.010465+0000: peer_request:mds.1:1879 currently dispatched 2023-08-25T02:33:25.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:25 smithi122 ceph-mon[100457]: pgmap v212: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 14 op/s 2023-08-25T02:33:26.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:26 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:26.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:26 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:26.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:26 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:26.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:26 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:26.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:26 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:26.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:26 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:27.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:27 smithi089 ceph-mon[98732]: pgmap v213: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:27.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:27 smithi121 ceph-mon[100099]: pgmap v213: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:27.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:27 smithi122 ceph-mon[100457]: pgmap v213: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 KiB/s wr, 17 op/s 2023-08-25T02:33:28.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: 1658 slow requests, 5 included below; oldest blocked for > 331.741798 secs 2023-08-25T02:33:28.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: slow request 330.831010 seconds old, received at 2023-08-25T02:27:56.531645+0000: client_request(mds.1:2029 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: slow request 330.801623 seconds old, received at 2023-08-25T02:27:56.561033+0000: client_request(mds.1:2061 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: slow request 330.772210 seconds old, received at 2023-08-25T02:27:56.590446+0000: client_request(mds.1:2093 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: slow request 330.742481 seconds old, received at 2023-08-25T02:27:56.620174+0000: client_request(mds.1:2125 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:28 smithi089 ceph-mon[98732]: slow request 330.712834 seconds old, received at 2023-08-25T02:27:56.649821+0000: client_request(mds.1:2157 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: 1658 slow requests, 5 included below; oldest blocked for > 331.741798 secs 2023-08-25T02:33:28.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: slow request 330.831010 seconds old, received at 2023-08-25T02:27:56.531645+0000: client_request(mds.1:2029 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: slow request 330.801623 seconds old, received at 2023-08-25T02:27:56.561033+0000: client_request(mds.1:2061 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: slow request 330.772210 seconds old, received at 2023-08-25T02:27:56.590446+0000: client_request(mds.1:2093 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: slow request 330.742481 seconds old, received at 2023-08-25T02:27:56.620174+0000: client_request(mds.1:2125 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:28 smithi121 ceph-mon[100099]: slow request 330.712834 seconds old, received at 2023-08-25T02:27:56.649821+0000: client_request(mds.1:2157 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: 1658 slow requests, 5 included below; oldest blocked for > 331.741798 secs 2023-08-25T02:33:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: slow request 330.831010 seconds old, received at 2023-08-25T02:27:56.531645+0000: client_request(mds.1:2029 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: slow request 330.801623 seconds old, received at 2023-08-25T02:27:56.561033+0000: client_request(mds.1:2061 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: slow request 330.772210 seconds old, received at 2023-08-25T02:27:56.590446+0000: client_request(mds.1:2093 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: slow request 330.742481 seconds old, received at 2023-08-25T02:27:56.620174+0000: client_request(mds.1:2125 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:28.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:28 smithi122 ceph-mon[100457]: slow request 330.712834 seconds old, received at 2023-08-25T02:27:56.649821+0000: client_request(mds.1:2157 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:29.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:29 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:29 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:29.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:29 smithi089 ceph-mon[98732]: pgmap v214: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:29.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:29 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:29.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:29 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:29.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:29 smithi121 ceph-mon[100099]: pgmap v214: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:29.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:29 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:29.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:29 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:29.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:29 smithi122 ceph-mon[100457]: pgmap v214: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:30.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: 1654 slow requests, 5 included below; oldest blocked for > 323.565276 secs 2023-08-25T02:33:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: slow request 323.377396 seconds old, received at 2023-08-25T02:28:06.018185+0000: peer_request:mds.1:1911 currently dispatched 2023-08-25T02:33:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: slow request 323.369579 seconds old, received at 2023-08-25T02:28:06.026002+0000: peer_request:mds.1:1943 currently dispatched 2023-08-25T02:33:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: slow request 323.361788 seconds old, received at 2023-08-25T02:28:06.033793+0000: peer_request:mds.1:1975 currently dispatched 2023-08-25T02:33:30.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: slow request 323.353926 seconds old, received at 2023-08-25T02:28:06.041655+0000: peer_request:mds.1:2007 currently dispatched 2023-08-25T02:33:30.581 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:30 smithi089 ceph-mon[98732]: slow request 323.346666 seconds old, received at 2023-08-25T02:28:06.048916+0000: peer_request:mds.1:2039 currently dispatched 2023-08-25T02:33:30.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: 1654 slow requests, 5 included below; oldest blocked for > 323.565276 secs 2023-08-25T02:33:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: slow request 323.377396 seconds old, received at 2023-08-25T02:28:06.018185+0000: peer_request:mds.1:1911 currently dispatched 2023-08-25T02:33:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: slow request 323.369579 seconds old, received at 2023-08-25T02:28:06.026002+0000: peer_request:mds.1:1943 currently dispatched 2023-08-25T02:33:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: slow request 323.361788 seconds old, received at 2023-08-25T02:28:06.033793+0000: peer_request:mds.1:1975 currently dispatched 2023-08-25T02:33:30.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: slow request 323.353926 seconds old, received at 2023-08-25T02:28:06.041655+0000: peer_request:mds.1:2007 currently dispatched 2023-08-25T02:33:30.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:30 smithi121 ceph-mon[100099]: slow request 323.346666 seconds old, received at 2023-08-25T02:28:06.048916+0000: peer_request:mds.1:2039 currently dispatched 2023-08-25T02:33:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: 1654 slow requests, 5 included below; oldest blocked for > 323.565276 secs 2023-08-25T02:33:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: slow request 323.377396 seconds old, received at 2023-08-25T02:28:06.018185+0000: peer_request:mds.1:1911 currently dispatched 2023-08-25T02:33:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: slow request 323.369579 seconds old, received at 2023-08-25T02:28:06.026002+0000: peer_request:mds.1:1943 currently dispatched 2023-08-25T02:33:30.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: slow request 323.361788 seconds old, received at 2023-08-25T02:28:06.033793+0000: peer_request:mds.1:1975 currently dispatched 2023-08-25T02:33:30.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: slow request 323.353926 seconds old, received at 2023-08-25T02:28:06.041655+0000: peer_request:mds.1:2007 currently dispatched 2023-08-25T02:33:30.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:30 smithi122 ceph-mon[100457]: slow request 323.346666 seconds old, received at 2023-08-25T02:28:06.048916+0000: peer_request:mds.1:2039 currently dispatched 2023-08-25T02:33:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:31 smithi089 ceph-mon[98732]: pgmap v215: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:31.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:31 smithi121 ceph-mon[100099]: pgmap v215: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:31.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:31 smithi122 ceph-mon[100457]: pgmap v215: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-08-25T02:33:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: pgmap v216: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 11 op/s 2023-08-25T02:33:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: 1647 slow requests, 5 included below; oldest blocked for > 337.007719 secs 2023-08-25T02:33:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: slow request 335.949476 seconds old, received at 2023-08-25T02:27:56.679100+0000: client_request(mds.1:2189 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: slow request 335.919817 seconds old, received at 2023-08-25T02:27:56.708759+0000: client_request(mds.1:2221 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: slow request 335.890489 seconds old, received at 2023-08-25T02:27:56.738087+0000: client_request(mds.1:2253 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: slow request 335.860451 seconds old, received at 2023-08-25T02:27:56.768125+0000: client_request(mds.1:2285 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:33 smithi089 ceph-mon[98732]: slow request 335.830834 seconds old, received at 2023-08-25T02:27:56.797742+0000: client_request(mds.1:2317 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: pgmap v216: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 11 op/s 2023-08-25T02:33:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: 1647 slow requests, 5 included below; oldest blocked for > 337.007719 secs 2023-08-25T02:33:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: slow request 335.949476 seconds old, received at 2023-08-25T02:27:56.679100+0000: client_request(mds.1:2189 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: slow request 335.919817 seconds old, received at 2023-08-25T02:27:56.708759+0000: client_request(mds.1:2221 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: slow request 335.890489 seconds old, received at 2023-08-25T02:27:56.738087+0000: client_request(mds.1:2253 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: slow request 335.860451 seconds old, received at 2023-08-25T02:27:56.768125+0000: client_request(mds.1:2285 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:33 smithi121 ceph-mon[100099]: slow request 335.830834 seconds old, received at 2023-08-25T02:27:56.797742+0000: client_request(mds.1:2317 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: pgmap v216: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 11 op/s 2023-08-25T02:33:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: 1647 slow requests, 5 included below; oldest blocked for > 337.007719 secs 2023-08-25T02:33:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: slow request 335.949476 seconds old, received at 2023-08-25T02:27:56.679100+0000: client_request(mds.1:2189 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: slow request 335.919817 seconds old, received at 2023-08-25T02:27:56.708759+0000: client_request(mds.1:2221 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: slow request 335.890489 seconds old, received at 2023-08-25T02:27:56.738087+0000: client_request(mds.1:2253 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: slow request 335.860451 seconds old, received at 2023-08-25T02:27:56.768125+0000: client_request(mds.1:2285 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:33.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:33 smithi122 ceph-mon[100457]: slow request 335.830834 seconds old, received at 2023-08-25T02:27:56.797742+0000: client_request(mds.1:2317 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: 1643 slow requests, 5 included below; oldest blocked for > 328.565388 secs 2023-08-25T02:33:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: slow request 328.339964 seconds old, received at 2023-08-25T02:28:06.055730+0000: peer_request:mds.1:2071 currently dispatched 2023-08-25T02:33:34.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: slow request 328.333037 seconds old, received at 2023-08-25T02:28:06.062656+0000: peer_request:mds.1:2103 currently dispatched 2023-08-25T02:33:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: slow request 328.325852 seconds old, received at 2023-08-25T02:28:06.069842+0000: peer_request:mds.1:2135 currently dispatched 2023-08-25T02:33:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: slow request 328.318597 seconds old, received at 2023-08-25T02:28:06.077097+0000: peer_request:mds.1:2167 currently dispatched 2023-08-25T02:33:34.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:34 smithi089 ceph-mon[98732]: slow request 328.311361 seconds old, received at 2023-08-25T02:28:06.084333+0000: peer_request:mds.1:2199 currently dispatched 2023-08-25T02:33:34.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:34.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: 1643 slow requests, 5 included below; oldest blocked for > 328.565388 secs 2023-08-25T02:33:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: slow request 328.339964 seconds old, received at 2023-08-25T02:28:06.055730+0000: peer_request:mds.1:2071 currently dispatched 2023-08-25T02:33:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: slow request 328.333037 seconds old, received at 2023-08-25T02:28:06.062656+0000: peer_request:mds.1:2103 currently dispatched 2023-08-25T02:33:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: slow request 328.325852 seconds old, received at 2023-08-25T02:28:06.069842+0000: peer_request:mds.1:2135 currently dispatched 2023-08-25T02:33:34.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: slow request 328.318597 seconds old, received at 2023-08-25T02:28:06.077097+0000: peer_request:mds.1:2167 currently dispatched 2023-08-25T02:33:34.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:34 smithi121 ceph-mon[100099]: slow request 328.311361 seconds old, received at 2023-08-25T02:28:06.084333+0000: peer_request:mds.1:2199 currently dispatched 2023-08-25T02:33:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: 1643 slow requests, 5 included below; oldest blocked for > 328.565388 secs 2023-08-25T02:33:34.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: slow request 328.339964 seconds old, received at 2023-08-25T02:28:06.055730+0000: peer_request:mds.1:2071 currently dispatched 2023-08-25T02:33:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: slow request 328.333037 seconds old, received at 2023-08-25T02:28:06.062656+0000: peer_request:mds.1:2103 currently dispatched 2023-08-25T02:33:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: slow request 328.325852 seconds old, received at 2023-08-25T02:28:06.069842+0000: peer_request:mds.1:2135 currently dispatched 2023-08-25T02:33:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: slow request 328.318597 seconds old, received at 2023-08-25T02:28:06.077097+0000: peer_request:mds.1:2167 currently dispatched 2023-08-25T02:33:34.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:34 smithi122 ceph-mon[100457]: slow request 328.311361 seconds old, received at 2023-08-25T02:28:06.084333+0000: peer_request:mds.1:2199 currently dispatched 2023-08-25T02:33:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:35 smithi089 ceph-mon[98732]: pgmap v217: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-08-25T02:33:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:35 smithi121 ceph-mon[100099]: pgmap v217: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-08-25T02:33:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:35 smithi122 ceph-mon[100457]: pgmap v217: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-08-25T02:33:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:37 smithi089 ceph-mon[98732]: pgmap v218: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-08-25T02:33:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:37 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:37.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:37 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:37 smithi121 ceph-mon[100099]: pgmap v218: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-08-25T02:33:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:37 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:37.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:37 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:37 smithi122 ceph-mon[100457]: pgmap v218: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-08-25T02:33:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:37 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:37.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:37 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:38.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:38 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:38.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:38 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:38.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:38 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:38.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:38 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:38 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:38.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:38 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: 1380 slow requests, 5 included below; oldest blocked for > 342.246242 secs 2023-08-25T02:33:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 341.039968 seconds old, received at 2023-08-25T02:27:56.827131+0000: client_request(mds.1:2349 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 341.009031 seconds old, received at 2023-08-25T02:27:56.858068+0000: client_request(mds.1:2381 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 340.977184 seconds old, received at 2023-08-25T02:27:56.889916+0000: client_request(mds.1:2413 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 340.945347 seconds old, received at 2023-08-25T02:27:56.921753+0000: client_request(mds.1:2445 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 340.913396 seconds old, received at 2023-08-25T02:27:56.953704+0000: client_request(mds.1:2477 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: pgmap v219: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:39.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: 703 slow requests, 5 included below; oldest blocked for > 333.565534 secs 2023-08-25T02:33:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 333.224761 seconds old, received at 2023-08-25T02:28:06.171078+0000: peer_request:mds.1:2583 currently dispatched 2023-08-25T02:33:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 333.218564 seconds old, received at 2023-08-25T02:28:06.177275+0000: peer_request:mds.1:2615 currently dispatched 2023-08-25T02:33:39.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 333.212294 seconds old, received at 2023-08-25T02:28:06.183545+0000: peer_request:mds.1:2647 currently dispatched 2023-08-25T02:33:39.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 333.205945 seconds old, received at 2023-08-25T02:28:06.189894+0000: peer_request:mds.1:2679 currently dispatched 2023-08-25T02:33:39.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:39 smithi089 ceph-mon[98732]: slow request 333.199654 seconds old, received at 2023-08-25T02:28:06.196184+0000: peer_request:mds.1:2711 currently dispatched 2023-08-25T02:33:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: 1380 slow requests, 5 included below; oldest blocked for > 342.246242 secs 2023-08-25T02:33:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 341.039968 seconds old, received at 2023-08-25T02:27:56.827131+0000: client_request(mds.1:2349 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 341.009031 seconds old, received at 2023-08-25T02:27:56.858068+0000: client_request(mds.1:2381 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 340.977184 seconds old, received at 2023-08-25T02:27:56.889916+0000: client_request(mds.1:2413 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 340.945347 seconds old, received at 2023-08-25T02:27:56.921753+0000: client_request(mds.1:2445 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 340.913396 seconds old, received at 2023-08-25T02:27:56.953704+0000: client_request(mds.1:2477 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: pgmap v219: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:39.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:39.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: 703 slow requests, 5 included below; oldest blocked for > 333.565534 secs 2023-08-25T02:33:39.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 333.224761 seconds old, received at 2023-08-25T02:28:06.171078+0000: peer_request:mds.1:2583 currently dispatched 2023-08-25T02:33:39.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 333.218564 seconds old, received at 2023-08-25T02:28:06.177275+0000: peer_request:mds.1:2615 currently dispatched 2023-08-25T02:33:39.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 333.212294 seconds old, received at 2023-08-25T02:28:06.183545+0000: peer_request:mds.1:2647 currently dispatched 2023-08-25T02:33:39.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 333.205945 seconds old, received at 2023-08-25T02:28:06.189894+0000: peer_request:mds.1:2679 currently dispatched 2023-08-25T02:33:39.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:39 smithi121 ceph-mon[100099]: slow request 333.199654 seconds old, received at 2023-08-25T02:28:06.196184+0000: peer_request:mds.1:2711 currently dispatched 2023-08-25T02:33:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: 1380 slow requests, 5 included below; oldest blocked for > 342.246242 secs 2023-08-25T02:33:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 341.039968 seconds old, received at 2023-08-25T02:27:56.827131+0000: client_request(mds.1:2349 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 341.009031 seconds old, received at 2023-08-25T02:27:56.858068+0000: client_request(mds.1:2381 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 340.977184 seconds old, received at 2023-08-25T02:27:56.889916+0000: client_request(mds.1:2413 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 340.945347 seconds old, received at 2023-08-25T02:27:56.921753+0000: client_request(mds.1:2445 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 340.913396 seconds old, received at 2023-08-25T02:27:56.953704+0000: client_request(mds.1:2477 rename #0x10000000002/000000010000000100000007 #0x60c/1000000051f caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2023-08-25T02:33:39.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: pgmap v219: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: 703 slow requests, 5 included below; oldest blocked for > 333.565534 secs 2023-08-25T02:33:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 333.224761 seconds old, received at 2023-08-25T02:28:06.171078+0000: peer_request:mds.1:2583 currently dispatched 2023-08-25T02:33:39.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 333.218564 seconds old, received at 2023-08-25T02:28:06.177275+0000: peer_request:mds.1:2615 currently dispatched 2023-08-25T02:33:39.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 333.212294 seconds old, received at 2023-08-25T02:28:06.183545+0000: peer_request:mds.1:2647 currently dispatched 2023-08-25T02:33:39.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 333.205945 seconds old, received at 2023-08-25T02:28:06.189894+0000: peer_request:mds.1:2679 currently dispatched 2023-08-25T02:33:39.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:39 smithi122 ceph-mon[100457]: slow request 333.199654 seconds old, received at 2023-08-25T02:28:06.196184+0000: peer_request:mds.1:2711 currently dispatched 2023-08-25T02:33:41.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:41 smithi121 ceph-mon[100099]: pgmap v220: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:41 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:41.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:41 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:41 smithi122 ceph-mon[100457]: pgmap v220: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:41 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:41.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:41 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:42.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:41 smithi089 ceph-mon[98732]: pgmap v220: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-08-25T02:33:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:41 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:42.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:41 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:42.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:42 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:42.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:42 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:42.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:42 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:42.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:42 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:43.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:42 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:43.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:42 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:43 smithi121 ceph-mon[100099]: pgmap v221: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 12 op/s 2023-08-25T02:33:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:43 smithi122 ceph-mon[100457]: pgmap v221: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 12 op/s 2023-08-25T02:33:44.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:43 smithi089 ceph-mon[98732]: pgmap v221: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 12 op/s 2023-08-25T02:33:44.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:44 smithi121 ceph-mon[100099]: MDS health message cleared (mds.?): 1380 slow requests are blocked > 180 secs 2023-08-25T02:33:44.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:44 smithi122 ceph-mon[100457]: MDS health message cleared (mds.?): 1380 slow requests are blocked > 180 secs 2023-08-25T02:33:45.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:44 smithi089 ceph-mon[98732]: MDS health message cleared (mds.?): 1380 slow requests are blocked > 180 secs 2023-08-25T02:33:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: pgmap v222: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-08-25T02:33:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:33:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:45.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: MDS health message cleared (mds.?): 703 slow requests are blocked > 180 secs 2023-08-25T02:33:45.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:45 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:33:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: pgmap v222: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-08-25T02:33:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:33:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: MDS health message cleared (mds.?): 703 slow requests are blocked > 180 secs 2023-08-25T02:33:45.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:45 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:33:46.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: pgmap v222: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-08-25T02:33:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-08-25T02:33:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/3754052688,v1:172.21.15.122:6839/3754052688] up:active 2023-08-25T02:33:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:46.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: MDS health message cleared (mds.?): 703 slow requests are blocked > 180 secs 2023-08-25T02:33:46.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:45 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:33:46.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:46 smithi121 ceph-mon[100099]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-08-25T02:33:46.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:46 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:33:46.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:46 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:46.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:46 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:46.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:46 smithi122 ceph-mon[100457]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-08-25T02:33:46.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:46 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:33:46.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:46 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:46.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:46 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:47.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:46 smithi089 ceph-mon[98732]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-08-25T02:33:47.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:46 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:33:47.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:46 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6838/1864669733,v1:172.21.15.121:6839/1864669733] up:active 2023-08-25T02:33:47.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:46 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:33:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:47 smithi121 ceph-mon[100099]: pgmap v223: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2023-08-25T02:33:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:47 smithi122 ceph-mon[100457]: pgmap v223: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2023-08-25T02:33:48.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:47 smithi089 ceph-mon[98732]: pgmap v223: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2023-08-25T02:33:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:49 smithi121 ceph-mon[100099]: pgmap v224: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:49 smithi122 ceph-mon[100457]: pgmap v224: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:49 smithi089 ceph-mon[98732]: pgmap v224: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:51 smithi089 ceph-mon[98732]: pgmap v225: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:51 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:33:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:51 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:33:51.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:51 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:33:51.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:51 smithi121 ceph-mon[100099]: pgmap v225: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:51 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:33:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:51 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:33:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:51 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:33:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:51 smithi122 ceph-mon[100457]: pgmap v225: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-08-25T02:33:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:51 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:33:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:51 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:33:51.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:51 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:33:53.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:53 smithi089 ceph-mon[98732]: pgmap v226: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-08-25T02:33:53.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:53 smithi121 ceph-mon[100099]: pgmap v226: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-08-25T02:33:53.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:53 smithi122 ceph-mon[100457]: pgmap v226: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-08-25T02:33:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:55 smithi089 ceph-mon[98732]: pgmap v227: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-08-25T02:33:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:55 smithi121 ceph-mon[100099]: pgmap v227: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-08-25T02:33:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:55 smithi122 ceph-mon[100457]: pgmap v227: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2023-08-25T02:33:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:57 smithi089 ceph-mon[98732]: pgmap v228: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-08-25T02:33:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:57 smithi121 ceph-mon[100099]: pgmap v228: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-08-25T02:33:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:57 smithi122 ceph-mon[100457]: pgmap v228: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-08-25T02:33:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:33:59 smithi089 ceph-mon[98732]: pgmap v229: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:33:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:33:59 smithi121 ceph-mon[100099]: pgmap v229: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:33:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:33:59 smithi122 ceph-mon[100457]: pgmap v229: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:34:01.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:01 smithi089 ceph-mon[98732]: pgmap v230: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:34:01.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:01 smithi121 ceph-mon[100099]: pgmap v230: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:34:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:01 smithi122 ceph-mon[100457]: pgmap v230: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-08-25T02:34:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:03 smithi089 ceph-mon[98732]: pgmap v231: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-08-25T02:34:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:03 smithi121 ceph-mon[100099]: pgmap v231: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-08-25T02:34:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:03 smithi122 ceph-mon[100457]: pgmap v231: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-08-25T02:34:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:05 smithi089 ceph-mon[98732]: pgmap v232: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-08-25T02:34:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:05 smithi121 ceph-mon[100099]: pgmap v232: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-08-25T02:34:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:05 smithi122 ceph-mon[100457]: pgmap v232: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-08-25T02:34:07.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:07 smithi089 ceph-mon[98732]: pgmap v233: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-08-25T02:34:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:07 smithi121 ceph-mon[100099]: pgmap v233: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-08-25T02:34:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:07 smithi122 ceph-mon[100457]: pgmap v233: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-08-25T02:34:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:09 smithi089 ceph-mon[98732]: pgmap v234: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-08-25T02:34:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:09 smithi121 ceph-mon[100099]: pgmap v234: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-08-25T02:34:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:09 smithi122 ceph-mon[100457]: pgmap v234: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-08-25T02:34:10.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:10 smithi089 ceph-mon[98732]: osdmap e86: 12 total, 12 up, 12 in 2023-08-25T02:34:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:10 smithi121 ceph-mon[100099]: osdmap e86: 12 total, 12 up, 12 in 2023-08-25T02:34:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:10 smithi122 ceph-mon[100457]: osdmap e86: 12 total, 12 up, 12 in 2023-08-25T02:34:11.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:11 smithi089 ceph-mon[98732]: pgmap v236: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 61 KiB/s wr, 25 op/s 2023-08-25T02:34:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:11 smithi121 ceph-mon[100099]: pgmap v236: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 61 KiB/s wr, 25 op/s 2023-08-25T02:34:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:11 smithi122 ceph-mon[100457]: pgmap v236: 97 pgs: 97 active+clean; 7.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 61 KiB/s wr, 25 op/s 2023-08-25T02:34:13.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:13 smithi089 ceph-mon[98732]: pgmap v237: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:13 smithi121 ceph-mon[100099]: pgmap v237: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:13 smithi122 ceph-mon[100457]: pgmap v237: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:15.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:15 smithi089 ceph-mon[98732]: pgmap v238: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:15.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:15 smithi089 ceph-mon[98732]: osdmap e87: 12 total, 12 up, 12 in 2023-08-25T02:34:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:15 smithi121 ceph-mon[100099]: pgmap v238: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:15 smithi121 ceph-mon[100099]: osdmap e87: 12 total, 12 up, 12 in 2023-08-25T02:34:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:15 smithi122 ceph-mon[100457]: pgmap v238: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-08-25T02:34:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:15 smithi122 ceph-mon[100457]: osdmap e87: 12 total, 12 up, 12 in 2023-08-25T02:34:17.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:17 smithi089 ceph-mon[98732]: pgmap v240: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-08-25T02:34:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:17 smithi121 ceph-mon[100099]: pgmap v240: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-08-25T02:34:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:17 smithi122 ceph-mon[100457]: pgmap v240: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-08-25T02:34:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:19 smithi121 ceph-mon[100099]: pgmap v241: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2023-08-25T02:34:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:19 smithi122 ceph-mon[100457]: pgmap v241: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2023-08-25T02:34:20.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:19 smithi089 ceph-mon[98732]: pgmap v241: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2023-08-25T02:34:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:21 smithi089 ceph-mon[98732]: pgmap v242: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2023-08-25T02:34:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:21 smithi121 ceph-mon[100099]: pgmap v242: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2023-08-25T02:34:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:21 smithi122 ceph-mon[100457]: pgmap v242: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2023-08-25T02:34:23.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:23 smithi089 ceph-mon[98732]: pgmap v243: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:23 smithi121 ceph-mon[100099]: pgmap v243: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:23.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:23 smithi122 ceph-mon[100457]: pgmap v243: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:25.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:25 smithi089 ceph-mon[98732]: pgmap v244: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:25 smithi121 ceph-mon[100099]: pgmap v244: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:25 smithi122 ceph-mon[100457]: pgmap v244: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:34:27.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:27 smithi089 ceph-mon[98732]: pgmap v245: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:34:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:27 smithi121 ceph-mon[100099]: pgmap v245: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:34:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:27 smithi122 ceph-mon[100457]: pgmap v245: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:34:29.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:29 smithi089 ceph-mon[98732]: pgmap v246: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:29 smithi121 ceph-mon[100099]: pgmap v246: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:29 smithi122 ceph-mon[100457]: pgmap v246: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:31 smithi089 ceph-mon[98732]: pgmap v247: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:31.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:31 smithi121 ceph-mon[100099]: pgmap v247: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:31.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:31 smithi122 ceph-mon[100457]: pgmap v247: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:33 smithi089 ceph-mon[98732]: pgmap v248: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:33 smithi121 ceph-mon[100099]: pgmap v248: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:33 smithi122 ceph-mon[100457]: pgmap v248: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:35.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:35 smithi089 ceph-mon[98732]: pgmap v249: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:35 smithi121 ceph-mon[100099]: pgmap v249: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:35 smithi122 ceph-mon[100457]: pgmap v249: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:37.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:37 smithi089 ceph-mon[98732]: pgmap v250: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:37 smithi121 ceph-mon[100099]: pgmap v250: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:37 smithi122 ceph-mon[100457]: pgmap v250: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:39.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:39 smithi089 ceph-mon[98732]: pgmap v251: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:39 smithi121 ceph-mon[100099]: pgmap v251: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:39 smithi122 ceph-mon[100457]: pgmap v251: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:41 smithi089 ceph-mon[98732]: pgmap v252: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:41.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:41 smithi121 ceph-mon[100099]: pgmap v252: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:41.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:41 smithi122 ceph-mon[100457]: pgmap v252: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:43.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:43 smithi089 ceph-mon[98732]: pgmap v253: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:43 smithi121 ceph-mon[100099]: pgmap v253: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:43 smithi122 ceph-mon[100457]: pgmap v253: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:45.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:45 smithi089 ceph-mon[98732]: pgmap v254: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:45 smithi121 ceph-mon[100099]: pgmap v254: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:45 smithi122 ceph-mon[100457]: pgmap v254: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:47 smithi089 ceph-mon[98732]: pgmap v255: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:47.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:47 smithi121 ceph-mon[100099]: pgmap v255: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:47 smithi122 ceph-mon[100457]: pgmap v255: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:49.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:49 smithi089 ceph-mon[98732]: pgmap v256: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:49 smithi121 ceph-mon[100099]: pgmap v256: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:49 smithi122 ceph-mon[100457]: pgmap v256: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:51 smithi089 ceph-mon[98732]: pgmap v257: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:51 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:34:51.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:51 smithi121 ceph-mon[100099]: pgmap v257: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:51.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:51 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:34:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:51 smithi122 ceph-mon[100457]: pgmap v257: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:51 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:34:52.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:52 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:34:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:52 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:34:52.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:52 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:34:52.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:52 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:34:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:52 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:34:52.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:52 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:34:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:52 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:34:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:52 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:34:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:52 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:34:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:53 smithi089 ceph-mon[98732]: pgmap v258: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:53 smithi121 ceph-mon[100099]: pgmap v258: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:53 smithi122 ceph-mon[100457]: pgmap v258: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:55.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:55 smithi089 ceph-mon[98732]: pgmap v259: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:55 smithi121 ceph-mon[100099]: pgmap v259: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:55 smithi122 ceph-mon[100457]: pgmap v259: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:34:57.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:57 smithi089 ceph-mon[98732]: pgmap v260: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:57 smithi121 ceph-mon[100099]: pgmap v260: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:57.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:57 smithi122 ceph-mon[100457]: pgmap v260: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:34:59.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:34:59 smithi089 ceph-mon[98732]: pgmap v261: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:34:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:34:59 smithi121 ceph-mon[100099]: pgmap v261: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:34:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:34:59 smithi122 ceph-mon[100457]: pgmap v261: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:35:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:01 smithi089 ceph-mon[98732]: pgmap v262: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:35:01.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:01 smithi121 ceph-mon[100099]: pgmap v262: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:35:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:01 smithi122 ceph-mon[100457]: pgmap v262: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-08-25T02:35:03.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:03 smithi089 ceph-mon[98732]: pgmap v263: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-08-25T02:35:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:03 smithi121 ceph-mon[100099]: pgmap v263: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-08-25T02:35:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:03 smithi122 ceph-mon[100457]: pgmap v263: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-08-25T02:35:05.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:05 smithi089 ceph-mon[98732]: pgmap v264: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:05 smithi121 ceph-mon[100099]: pgmap v264: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:05 smithi122 ceph-mon[100457]: pgmap v264: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:07.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:07 smithi089 ceph-mon[98732]: pgmap v265: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:35:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:07 smithi121 ceph-mon[100099]: pgmap v265: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:35:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:07 smithi122 ceph-mon[100457]: pgmap v265: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:35:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:09 smithi121 ceph-mon[100099]: pgmap v266: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:09 smithi122 ceph-mon[100457]: pgmap v266: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:10.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:09 smithi089 ceph-mon[98732]: pgmap v266: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:35:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:10 smithi121 ceph-mon[100099]: osdmap e88: 12 total, 12 up, 12 in 2023-08-25T02:35:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:10 smithi122 ceph-mon[100457]: osdmap e88: 12 total, 12 up, 12 in 2023-08-25T02:35:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:10 smithi089 ceph-mon[98732]: osdmap e88: 12 total, 12 up, 12 in 2023-08-25T02:35:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:11 smithi121 ceph-mon[100099]: pgmap v268: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:35:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:11 smithi122 ceph-mon[100457]: pgmap v268: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:35:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:11 smithi089 ceph-mon[98732]: pgmap v268: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:35:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:13 smithi121 ceph-mon[100099]: pgmap v269: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:13 smithi122 ceph-mon[100457]: pgmap v269: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:13 smithi089 ceph-mon[98732]: pgmap v269: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:15 smithi121 ceph-mon[100099]: pgmap v270: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:15 smithi121 ceph-mon[100099]: osdmap e89: 12 total, 12 up, 12 in 2023-08-25T02:35:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:15 smithi122 ceph-mon[100457]: pgmap v270: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:15 smithi122 ceph-mon[100457]: osdmap e89: 12 total, 12 up, 12 in 2023-08-25T02:35:16.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:15 smithi089 ceph-mon[98732]: pgmap v270: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:16.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:15 smithi089 ceph-mon[98732]: osdmap e89: 12 total, 12 up, 12 in 2023-08-25T02:35:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:17 smithi121 ceph-mon[100099]: pgmap v272: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:35:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:17 smithi122 ceph-mon[100457]: pgmap v272: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:35:18.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:17 smithi089 ceph-mon[98732]: pgmap v272: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:35:19.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:19 smithi121 ceph-mon[100099]: pgmap v273: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:35:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:19 smithi122 ceph-mon[100457]: pgmap v273: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:35:20.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:19 smithi089 ceph-mon[98732]: pgmap v273: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:35:20.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:20 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:20.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:20 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:21.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:20 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:21.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:21 smithi121 ceph-mon[100099]: pgmap v274: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:21.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:21 smithi122 ceph-mon[100457]: pgmap v274: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:22.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:21 smithi089 ceph-mon[98732]: pgmap v274: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:35:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:23 smithi121 ceph-mon[100099]: pgmap v275: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:23.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:23 smithi122 ceph-mon[100457]: pgmap v275: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:24.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:23 smithi089 ceph-mon[98732]: pgmap v275: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:25 smithi121 ceph-mon[100099]: pgmap v276: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:25 smithi122 ceph-mon[100457]: pgmap v276: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:26.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:25 smithi089 ceph-mon[98732]: pgmap v276: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:35:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:27 smithi121 ceph-mon[100099]: pgmap v277: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:35:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:27 smithi122 ceph-mon[100457]: pgmap v277: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:35:28.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:27 smithi089 ceph-mon[98732]: pgmap v277: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:35:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:29 smithi121 ceph-mon[100099]: pgmap v278: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:29 smithi122 ceph-mon[100457]: pgmap v278: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:30.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:29 smithi089 ceph-mon[98732]: pgmap v278: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:31 smithi089 ceph-mon[98732]: pgmap v279: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:31.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:31 smithi121 ceph-mon[100099]: pgmap v279: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:31.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:31 smithi122 ceph-mon[100457]: pgmap v279: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:33.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:33 smithi089 ceph-mon[98732]: pgmap v280: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:33 smithi121 ceph-mon[100099]: pgmap v280: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:33 smithi122 ceph-mon[100457]: pgmap v280: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:35.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:35 smithi121 ceph-mon[100099]: pgmap v281: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:35.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:35 smithi122 ceph-mon[100457]: pgmap v281: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:36.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:35 smithi089 ceph-mon[98732]: pgmap v281: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:37.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:37 smithi121 ceph-mon[100099]: pgmap v282: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:37.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:37 smithi122 ceph-mon[100457]: pgmap v282: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:38.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:37 smithi089 ceph-mon[98732]: pgmap v282: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:39 smithi121 ceph-mon[100099]: pgmap v283: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:39 smithi122 ceph-mon[100457]: pgmap v283: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:40.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:39 smithi089 ceph-mon[98732]: pgmap v283: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:41 smithi089 ceph-mon[98732]: pgmap v284: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:41.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:41 smithi121 ceph-mon[100099]: pgmap v284: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:41.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:41 smithi122 ceph-mon[100457]: pgmap v284: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:43 smithi121 ceph-mon[100099]: pgmap v285: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:43 smithi122 ceph-mon[100457]: pgmap v285: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:44.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:43 smithi089 ceph-mon[98732]: pgmap v285: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:45 smithi121 ceph-mon[100099]: pgmap v286: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:45 smithi122 ceph-mon[100457]: pgmap v286: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:46.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:45 smithi089 ceph-mon[98732]: pgmap v286: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:47.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:47 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:47.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:47 smithi089 ceph-mon[98732]: pgmap v287: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:47.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:47 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:47.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:47 smithi121 ceph-mon[100099]: pgmap v287: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:47.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:47 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:47.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:47 smithi122 ceph-mon[100457]: pgmap v287: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:49 smithi121 ceph-mon[100099]: pgmap v288: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:49 smithi122 ceph-mon[100457]: pgmap v288: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:49 smithi089 ceph-mon[98732]: pgmap v288: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:51.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:51 smithi089 ceph-mon[98732]: pgmap v289: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:51.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:51 smithi121 ceph-mon[100099]: pgmap v289: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:51 smithi122 ceph-mon[100457]: pgmap v289: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:52.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:52 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:35:52.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:52 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:35:52.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:52 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:35:53.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:53 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:35:53.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:53 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:35:53.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:53 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:35:53.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:53 smithi089 ceph-mon[98732]: pgmap v290: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:53.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:53 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:35:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:53 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:35:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:53 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:35:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:53 smithi121 ceph-mon[100099]: pgmap v290: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:53 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:35:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:53 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:35:53.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:53 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:35:53.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:53 smithi122 ceph-mon[100457]: pgmap v290: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:55.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:55 smithi121 ceph-mon[100099]: pgmap v291: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:55 smithi122 ceph-mon[100457]: pgmap v291: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:56.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:55 smithi089 ceph-mon[98732]: pgmap v291: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:57 smithi121 ceph-mon[100099]: pgmap v292: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:57.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:57 smithi122 ceph-mon[100457]: pgmap v292: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:58.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:57 smithi089 ceph-mon[98732]: pgmap v292: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:35:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:59 smithi121 ceph-mon[100099]: pgmap v293: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:59.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:35:59 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:35:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:59 smithi122 ceph-mon[100457]: pgmap v293: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:35:59.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:35:59 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:00.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:59 smithi089 ceph-mon[98732]: pgmap v293: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:00.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:35:59 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:01 smithi089 ceph-mon[98732]: pgmap v294: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:01.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:01 smithi121 ceph-mon[100099]: pgmap v294: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:01 smithi122 ceph-mon[100457]: pgmap v294: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:03 smithi121 ceph-mon[100099]: pgmap v295: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:03 smithi122 ceph-mon[100457]: pgmap v295: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:04.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:03 smithi089 ceph-mon[98732]: pgmap v295: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:05 smithi121 ceph-mon[100099]: pgmap v296: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:05.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:05 smithi122 ceph-mon[100457]: pgmap v296: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:06.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:05 smithi089 ceph-mon[98732]: pgmap v296: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:07 smithi121 ceph-mon[100099]: pgmap v297: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:07 smithi122 ceph-mon[100457]: pgmap v297: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:08.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:07 smithi089 ceph-mon[98732]: pgmap v297: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:36:09.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:09 smithi121 ceph-mon[100099]: pgmap v298: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:09.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:09 smithi122 ceph-mon[100457]: pgmap v298: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:10.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:09 smithi089 ceph-mon[98732]: pgmap v298: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:36:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:10 smithi121 ceph-mon[100099]: osdmap e90: 12 total, 12 up, 12 in 2023-08-25T02:36:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:10 smithi122 ceph-mon[100457]: osdmap e90: 12 total, 12 up, 12 in 2023-08-25T02:36:11.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:10 smithi089 ceph-mon[98732]: osdmap e90: 12 total, 12 up, 12 in 2023-08-25T02:36:11.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:11 smithi121 ceph-mon[100099]: pgmap v300: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:36:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:11 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:11 smithi122 ceph-mon[100457]: pgmap v300: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:36:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:11 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:11 smithi089 ceph-mon[98732]: pgmap v300: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:36:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:11 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:13.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:13 smithi121 ceph-mon[100099]: pgmap v301: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:13.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:13 smithi122 ceph-mon[100457]: pgmap v301: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:14.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:13 smithi089 ceph-mon[98732]: pgmap v301: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:14 smithi121 ceph-mon[100099]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:14.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:14 smithi122 ceph-mon[100457]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:15.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:14 smithi089 ceph-mon[98732]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-08-25T02:36:15.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:15 smithi121 ceph-mon[100099]: pgmap v302: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:15.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:15 smithi121 ceph-mon[100099]: osdmap e91: 12 total, 12 up, 12 in 2023-08-25T02:36:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:15 smithi122 ceph-mon[100457]: pgmap v302: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:15.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:15 smithi122 ceph-mon[100457]: osdmap e91: 12 total, 12 up, 12 in 2023-08-25T02:36:16.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:15 smithi089 ceph-mon[98732]: pgmap v302: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:16.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:15 smithi089 ceph-mon[98732]: osdmap e91: 12 total, 12 up, 12 in 2023-08-25T02:36:17.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:17 smithi121 ceph-mon[100099]: pgmap v304: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:36:17.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:17 smithi122 ceph-mon[100457]: pgmap v304: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:36:18.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:17 smithi089 ceph-mon[98732]: pgmap v304: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-08-25T02:36:19.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:19 smithi121 ceph-mon[100099]: pgmap v305: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:36:19.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:19 smithi122 ceph-mon[100457]: pgmap v305: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:36:20.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:19 smithi089 ceph-mon[98732]: pgmap v305: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2023-08-25T02:36:21.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:21 smithi089 ceph-mon[98732]: pgmap v306: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:21.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:21 smithi121 ceph-mon[100099]: pgmap v306: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:21.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:21 smithi122 ceph-mon[100457]: pgmap v306: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-08-25T02:36:23.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:23 smithi121 ceph-mon[100099]: pgmap v307: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:23.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:23 smithi122 ceph-mon[100457]: pgmap v307: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:24.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:23 smithi089 ceph-mon[98732]: pgmap v307: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:25.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:25 smithi121 ceph-mon[100099]: pgmap v308: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:25.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:25 smithi122 ceph-mon[100457]: pgmap v308: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:26.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:25 smithi089 ceph-mon[98732]: pgmap v308: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-08-25T02:36:27.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:27 smithi121 ceph-mon[100099]: pgmap v309: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:36:27.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:27 smithi122 ceph-mon[100457]: pgmap v309: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:36:28.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:27 smithi089 ceph-mon[98732]: pgmap v309: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2023-08-25T02:36:29.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:29 smithi121 ceph-mon[100099]: pgmap v310: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:29.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:29 smithi122 ceph-mon[100457]: pgmap v310: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:30.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:29 smithi089 ceph-mon[98732]: pgmap v310: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:31.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:31 smithi089 ceph-mon[98732]: pgmap v311: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:31.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:31 smithi121 ceph-mon[100099]: pgmap v311: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:31.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:31 smithi122 ceph-mon[100457]: pgmap v311: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:32.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-08-25T02:36:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 -- 172.21.15.89:0/1482269867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c106920 msgr2=0x7f1f3c106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 --2- 172.21.15.89:0/1482269867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c106920 0x7f1f3c106d00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009990 tx=0x7f1f30009ca0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 -- 172.21.15.89:0/1482269867 shutdown_connections 2023-08-25T02:36:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 --2- 172.21.15.89:0/1482269867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c069660 0x7f1f3c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 --2- 172.21.15.89:0/1482269867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c106920 0x7f1f3c106d00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 --2- 172.21.15.89:0/1482269867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c068ca0 0x7f1f3c069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 -- 172.21.15.89:0/1482269867 >> 172.21.15.89:0/1482269867 conn(0x7f1f3c076920 msgr2=0x7f1f3c076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 -- 172.21.15.89:0/1482269867 shutdown_connections 2023-08-25T02:36:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.501+0000 7f1f41381700 1 -- 172.21.15.89:0/1482269867 wait complete. 2023-08-25T02:36:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 Processor -- start 2023-08-25T02:36:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 -- start start 2023-08-25T02:36:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c068ca0 0x7f1f3c0ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c106920 0x7f1f3c1a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1a50f0 con 0x7f1f3c069660 2023-08-25T02:36:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1a5230 con 0x7f1f3c068ca0 2023-08-25T02:36:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f41381700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1a53a0 con 0x7f1f3c106920 2023-08-25T02:36:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49392/0 (socket says 172.21.15.89:49392) 2023-08-25T02:36:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 -- 172.21.15.89:0/4082134743 learned_addr learned my addr 172.21.15.89:0/4082134743 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3affd700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c068ca0 0x7f1f3c0ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3b7fe700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c106920 0x7f1f3c1a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 -- 172.21.15.89:0/4082134743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c106920 msgr2=0x7f1f3c1a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c106920 0x7f1f3c1a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 -- 172.21.15.89:0/4082134743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c068ca0 msgr2=0x7f1f3c0ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c068ca0 0x7f1f3c0ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.502+0000 7f1f3a7fc700 1 -- 172.21.15.89:0/4082134743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300095f0 con 0x7f1f3c069660 2023-08-25T02:36:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f3a7fc700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00d950 tx=0x7f1f2c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00f890 con 0x7f1f3c069660 2023-08-25T02:36:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f2c010460 con 0x7f1f3c069660 2023-08-25T02:36:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c1a5680 con 0x7f1f3c069660 2023-08-25T02:36:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00e610 con 0x7f1f3c069660 2023-08-25T02:36:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.503+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c1a5be0 con 0x7f1f3c069660 2023-08-25T02:36:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.504+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c077440 con 0x7f1f3c069660 2023-08-25T02:36:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.505+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f1f2c0108d0 con 0x7f1f3c069660 2023-08-25T02:36:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.505+0000 7f1f23fff700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 0x7f1f240a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.505+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f1f2c0cb3b0 con 0x7f1f3c069660 2023-08-25T02:36:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.505+0000 7f1f3affd700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 0x7f1f240a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.506+0000 7f1f3affd700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 0x7f1f240a2600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000b2e0 tx=0x7f1f30005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:32.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.510+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f1f2c094dd0 con 0x7f1f3c069660 2023-08-25T02:36:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.727+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f3c110840 con 0x7f1f240a0140 2023-08-25T02:36:32.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.735+0000 7f1f23fff700 1 -- 172.21.15.89:0/4082134743 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7f1f3c110840 con 0x7f1f240a0140 2023-08-25T02:36:32.736 INFO:teuthology.orchestra.run.smithi089.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-08-25T00:00:00", "created": "2023-08-25T02:26:31", "first": "2023-08-25T02:27:00", "last": "2023-08-25T02:36:00", "last_pruned": "2023-08-25T02:36:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-08-25T02:36:32.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 msgr2=0x7f1f240a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 0x7f1f240a2600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000b2e0 tx=0x7f1f30005fd0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 msgr2=0x7f1f3c0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00d950 tx=0x7f1f2c00dc60 comp rx=0 tx=0).stop 2023-08-25T02:36:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 shutdown_connections 2023-08-25T02:36:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f1f240a0140 0x7f1f240a2600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f3c068ca0 0x7f1f3c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f3c106920 0x7f1f3c1a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 --2- 172.21.15.89:0/4082134743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3c069660 0x7f1f3c0fff80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 >> 172.21.15.89:0/4082134743 conn(0x7f1f3c076920 msgr2=0x7f1f3c0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 shutdown_connections 2023-08-25T02:36:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.737+0000 7f1f41381700 1 -- 172.21.15.89:0/4082134743 wait complete. 2023-08-25T02:36:32.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-08-25T02:36:32.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 -- 172.21.15.89:0/3815575491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8420100170 msgr2=0x7f842010c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 --2- 172.21.15.89:0/3815575491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8420100170 0x7f842010c680 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f841800b0a0 tx=0x7f841800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 -- 172.21.15.89:0/3815575491 shutdown_connections 2023-08-25T02:36:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 --2- 172.21.15.89:0/3815575491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8420100170 0x7f842010c680 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 --2- 172.21.15.89:0/3815575491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84201057c0 0x7f8420105ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 --2- 172.21.15.89:0/3815575491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 0x7f84200ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.870+0000 7f8425606700 1 -- 172.21.15.89:0/3815575491 >> 172.21.15.89:0/3815575491 conn(0x7f84200fb260 msgr2=0x7f84200fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 -- 172.21.15.89:0/3815575491 shutdown_connections 2023-08-25T02:36:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 -- 172.21.15.89:0/3815575491 wait complete. 2023-08-25T02:36:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 Processor -- start 2023-08-25T02:36:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 -- start start 2023-08-25T02:36:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 0x7f8420195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8420100170 0x7f8420195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.871+0000 7f8425606700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842019a920 con 0x7f84201057c0 2023-08-25T02:36:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8425606700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842019aa90 con 0x7f84200ff7b0 2023-08-25T02:36:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8425606700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842019ac50 con 0x7f8420100170 2023-08-25T02:36:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49408/0 (socket says 172.21.15.89:49408) 2023-08-25T02:36:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 -- 172.21.15.89:0/4181773120 learned_addr learned my addr 172.21.15.89:0/4181773120 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841effd700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 0x7f8420195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841e7fc700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8420100170 0x7f8420195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 -- 172.21.15.89:0/4181773120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8420100170 msgr2=0x7f8420195cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8420100170 0x7f8420195cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 -- 172.21.15.89:0/4181773120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 msgr2=0x7f8420195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 0x7f8420195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 -- 172.21.15.89:0/4181773120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418009d00 con 0x7f84201057c0 2023-08-25T02:36:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f841f7fe700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f841800b830 tx=0x7f8418009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841800e040 con 0x7f84201057c0 2023-08-25T02:36:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842019aed0 con 0x7f84201057c0 2023-08-25T02:36:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8418003c00 con 0x7f84201057c0 2023-08-25T02:36:32.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418005590 con 0x7f84201057c0 2023-08-25T02:36:32.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.872+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84201a27e0 con 0x7f84201057c0 2023-08-25T02:36:32.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.873+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84200feef0 con 0x7f84201057c0 2023-08-25T02:36:32.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.874+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f84180056f0 con 0x7f84201057c0 2023-08-25T02:36:32.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.875+0000 7f8407fff700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 0x7f84080a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:32.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.875+0000 7f841effd700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 0x7f84080a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:32.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.875+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f84180cd440 con 0x7f84201057c0 2023-08-25T02:36:32.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.875+0000 7f841effd700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 0x7f84080a2580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f841000abd0 tx=0x7f841000a380 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:32.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:32.878+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8418096f80 con 0x7f84201057c0 2023-08-25T02:36:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.097+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 --> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f842010f6d0 con 0x7f84080a00c0 2023-08-25T02:36:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.104+0000 7f8407fff700 1 -- 172.21.15.89:0/4181773120 <== mgr.24889 v2:172.21.15.89:6800/1258779165 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f842010f6d0 con 0x7f84080a00c0 2023-08-25T02:36:33.105 INFO:teuthology.orchestra.run.smithi089.stdout:/ 1M 6M3h 2023-08-25T02:36:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 msgr2=0x7f84080a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 0x7f84080a2580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f841000abd0 tx=0x7f841000a380 comp rx=0 tx=0).stop 2023-08-25T02:36:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 msgr2=0x7f842019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f841800b830 tx=0x7f8418009c70 comp rx=0 tx=0).stop 2023-08-25T02:36:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 shutdown_connections 2023-08-25T02:36:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f84080a00c0 0x7f84080a2580 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84200ff7b0 0x7f8420195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84201057c0 0x7f842019a350 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 --2- 172.21.15.89:0/4181773120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8420100170 0x7f8420195cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.106+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 >> 172.21.15.89:0/4181773120 conn(0x7f84200fb260 msgr2=0x7f84201032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.107+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 shutdown_connections 2023-08-25T02:36:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.107+0000 7f8425606700 1 -- 172.21.15.89:0/4181773120 wait complete. 2023-08-25T02:36:33.120 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-08-25T02:36:33.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:36:33.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 -- 172.21.15.89:0/1273252233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea481068b0 msgr2=0x7fea48106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 --2- 172.21.15.89:0/1273252233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea481068b0 0x7fea48106c90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009990 tx=0x7fea3c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 -- 172.21.15.89:0/1273252233 shutdown_connections 2023-08-25T02:36:33.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 --2- 172.21.15.89:0/1273252233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea48101260 0x7fea4810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 --2- 172.21.15.89:0/1273252233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea481068b0 0x7fea48106c90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 --2- 172.21.15.89:0/1273252233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481008a0 0x7fea48100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 -- 172.21.15.89:0/1273252233 >> 172.21.15.89:0/1273252233 conn(0x7fea480fc3b0 msgr2=0x7fea480fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 -- 172.21.15.89:0/1273252233 shutdown_connections 2023-08-25T02:36:33.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.252+0000 7fea4d55f700 1 -- 172.21.15.89:0/1273252233 wait complete. 2023-08-25T02:36:33.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 Processor -- start 2023-08-25T02:36:33.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 -- start start 2023-08-25T02:36:33.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea481008a0 0x7fea4819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 0x7fea48196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48196dd0 con 0x7fea48101260 2023-08-25T02:36:33.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48196f40 con 0x7fea481068b0 2023-08-25T02:36:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea4d55f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48197100 con 0x7fea481008a0 2023-08-25T02:36:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49420/0 (socket says 172.21.15.89:49420) 2023-08-25T02:36:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 -- 172.21.15.89:0/419741349 learned_addr learned my addr 172.21.15.89:0/419741349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea46ffd700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea481008a0 0x7fea4819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea477fe700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 0x7fea48196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 -- 172.21.15.89:0/419741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea481008a0 msgr2=0x7fea4819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea481008a0 0x7fea4819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 -- 172.21.15.89:0/419741349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 msgr2=0x7fea48196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 0x7fea48196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea467fc700 1 -- 172.21.15.89:0/419741349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0095f0 con 0x7fea48101260 2023-08-25T02:36:33.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.253+0000 7fea477fe700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 0x7fea48196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:36:33.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea467fc700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e9a0 tx=0x7fea3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38009790 con 0x7fea48101260 2023-08-25T02:36:33.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea481973e0 con 0x7fea48101260 2023-08-25T02:36:33.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea3800cbd0 con 0x7fea48101260 2023-08-25T02:36:33.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38018680 con 0x7fea48101260 2023-08-25T02:36:33.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.254+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea481a36e0 con 0x7fea48101260 2023-08-25T02:36:33.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.255+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea48067b60 con 0x7fea48101260 2023-08-25T02:36:33.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.256+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fea3800cd40 con 0x7fea48101260 2023-08-25T02:36:33.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.256+0000 7fea2ffff700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 0x7fea300a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.256+0000 7fea46ffd700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 0x7fea300a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.256+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7fea38014070 con 0x7fea48101260 2023-08-25T02:36:33.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.257+0000 7fea46ffd700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 0x7fea300a25c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00b2e0 tx=0x7fea3c005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.260+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fea380955f0 con 0x7fea48101260 2023-08-25T02:36:33.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.451+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea48110910 con 0x7fea48101260 2023-08-25T02:36:33.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.452+0000 7fea2ffff700 1 -- 172.21.15.89:0/419741349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23043 (secure 0 0 0) 0x7fea38098c10 con 0x7fea48101260 2023-08-25T02:36:33.454 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:36:33.454 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:36:15.259886+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":90,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/989947782":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/1951762039":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3424586982":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/2658035648":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:36:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 msgr2=0x7fea300a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 0x7fea300a25c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00b2e0 tx=0x7fea3c005fd0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 msgr2=0x7fea4819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e9a0 tx=0x7fea3800ecb0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 shutdown_connections 2023-08-25T02:36:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fea300a0100 0x7fea300a25c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea481068b0 0x7fea48196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea481008a0 0x7fea4819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 --2- 172.21.15.89:0/419741349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea48101260 0x7fea4819cc30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.454+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 >> 172.21.15.89:0/419741349 conn(0x7fea480fc3b0 msgr2=0x7fea481043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.455+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 shutdown_connections 2023-08-25T02:36:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.455+0000 7fea4d55f700 1 -- 172.21.15.89:0/419741349 wait complete. 2023-08-25T02:36:33.468 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 90, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:36:33.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:36:33.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 -- 172.21.15.89:0/1329597471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb0105c40 msgr2=0x7f9bb010d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/1329597471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb0105c40 0x7f9bb010d7f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800b0a0 tx=0x7f9ba800b3b0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 -- 172.21.15.89:0/1329597471 shutdown_connections 2023-08-25T02:36:33.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/1329597471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb0105c40 0x7f9bb010d7f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/1329597471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb00ff370 0x7f9bb00ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/1329597471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ffd20 0x7f9bb0105560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 -- 172.21.15.89:0/1329597471 >> 172.21.15.89:0/1329597471 conn(0x7f9bb0074ac0 msgr2=0x7f9bb0074ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 -- 172.21.15.89:0/1329597471 shutdown_connections 2023-08-25T02:36:33.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.591+0000 7f9bb52b0700 1 -- 172.21.15.89:0/1329597471 wait complete. 2023-08-25T02:36:33.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 Processor -- start 2023-08-25T02:36:33.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 -- start start 2023-08-25T02:36:33.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ff370 0x7f9bb019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 0x7f9bb019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb0197370 con 0x7f9bb01968f0 2023-08-25T02:36:33.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb01974b0 con 0x7f9bb00ff370 2023-08-25T02:36:33.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9bb52b0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb0197670 con 0x7f9bb0105c40 2023-08-25T02:36:33.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9baf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9baf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49436/0 (socket says 172.21.15.89:49436) 2023-08-25T02:36:33.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.592+0000 7f9baf7fe700 1 -- 172.21.15.89:0/4098694009 learned_addr learned my addr 172.21.15.89:0/4098694009 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:33.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baeffd700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ff370 0x7f9bb019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9bae7fc700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 0x7f9bb019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 -- 172.21.15.89:0/4098694009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 msgr2=0x7f9bb019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 0x7f9bb019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 -- 172.21.15.89:0/4098694009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ff370 msgr2=0x7f9bb019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ff370 0x7f9bb019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 -- 172.21.15.89:0/4098694009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba8009d00 con 0x7f9bb01968f0 2023-08-25T02:36:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9bae7fc700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 0x7f9bb019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:36:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9baf7fe700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800b780 tx=0x7f9ba8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800e040 con 0x7f9bb01968f0 2023-08-25T02:36:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ba8004c50 con 0x7f9bb01968f0 2023-08-25T02:36:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb01a3690 con 0x7f9bb01968f0 2023-08-25T02:36:33.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80052a0 con 0x7f9bb01968f0 2023-08-25T02:36:33.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.593+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb01a3b70 con 0x7f9bb01968f0 2023-08-25T02:36:33.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.594+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb0067af0 con 0x7f9bb01968f0 2023-08-25T02:36:33.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.595+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f9ba801d470 con 0x7f9bb01968f0 2023-08-25T02:36:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.596+0000 7f9ba7fff700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 0x7f9b9c0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.596+0000 7f9baeffd700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 0x7f9b9c0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.596+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f9ba80ccbd0 con 0x7f9bb01968f0 2023-08-25T02:36:33.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.596+0000 7f9baeffd700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 0x7f9b9c0a2580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009bf0 tx=0x7f9ba0009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.603+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f9ba8096680 con 0x7f9bb01968f0 2023-08-25T02:36:33.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.793+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bb00fff00 con 0x7f9bb01968f0 2023-08-25T02:36:33.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.793+0000 7f9ba7fff700 1 -- 172.21.15.89:0/4098694009 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ba8099ca0 con 0x7f9bb01968f0 2023-08-25T02:36:33.795 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:36:33.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.795+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 msgr2=0x7f9b9c0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.795+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 0x7f9b9c0a2580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009bf0 tx=0x7f9ba0009260 comp rx=0 tx=0).stop 2023-08-25T02:36:33.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 msgr2=0x7f9bb0196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800b780 tx=0x7f9ba8007a40 comp rx=0 tx=0).stop 2023-08-25T02:36:33.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 shutdown_connections 2023-08-25T02:36:33.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f9b9c0a00c0 0x7f9b9c0a2580 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb00ff370 0x7f9bb019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bb01968f0 0x7f9bb0196da0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 --2- 172.21.15.89:0/4098694009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bb0105c40 0x7f9bb019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 >> 172.21.15.89:0/4098694009 conn(0x7f9bb0074ac0 msgr2=0x7f9bb00fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 shutdown_connections 2023-08-25T02:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.796+0000 7f9bb52b0700 1 -- 172.21.15.89:0/4098694009 wait complete. 2023-08-25T02:36:33.808 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:33 smithi089 ceph-mon[98732]: pgmap v312: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:33.808 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:33 smithi089 ceph-mon[98732]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:33.808 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:33 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/419741349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:36:33.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:36:33.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 -- 172.21.15.89:0/363629229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc10aed0 msgr2=0x7f7afc10d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 --2- 172.21.15.89:0/363629229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc10aed0 0x7f7afc10d370 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7af400b0a0 tx=0x7f7af400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 -- 172.21.15.89:0/363629229 shutdown_connections 2023-08-25T02:36:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 --2- 172.21.15.89:0/363629229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc101b00 0x7f7afc101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 --2- 172.21.15.89:0/363629229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc10aed0 0x7f7afc10d370 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 --2- 172.21.15.89:0/363629229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc102420 0x7f7afc10a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 -- 172.21.15.89:0/363629229 >> 172.21.15.89:0/363629229 conn(0x7f7afc0fb2c0 msgr2=0x7f7afc0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:33.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 -- 172.21.15.89:0/363629229 shutdown_connections 2023-08-25T02:36:33.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.932+0000 7f7b01f83700 1 -- 172.21.15.89:0/363629229 wait complete. 2023-08-25T02:36:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 Processor -- start 2023-08-25T02:36:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 -- start start 2023-08-25T02:36:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc102420 0x7f7afc19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc10aed0 0x7f7afc1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc196f60 con 0x7f7afc101b00 2023-08-25T02:36:33.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc1970d0 con 0x7f7afc102420 2023-08-25T02:36:33.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7b01f83700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc197290 con 0x7f7afc10aed0 2023-08-25T02:36:33.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49446/0 (socket says 172.21.15.89:49446) 2023-08-25T02:36:33.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 -- 172.21.15.89:0/867576281 learned_addr learned my addr 172.21.15.89:0/867576281 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:33.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afaffd700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc102420 0x7f7afc19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afbfff700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc10aed0 0x7f7afc1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 -- 172.21.15.89:0/867576281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc10aed0 msgr2=0x7f7afc1968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc10aed0 0x7f7afc1968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 -- 172.21.15.89:0/867576281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc102420 msgr2=0x7f7afc19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:33.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc102420 0x7f7afc19cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:33.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.933+0000 7f7afb7fe700 1 -- 172.21.15.89:0/867576281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af4009d00 con 0x7f7afc101b00 2023-08-25T02:36:33.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7afb7fe700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c9b0 tx=0x7f7af000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007b00 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc197570 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7af0007c60 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0018660 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.934+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc1a3870 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.935+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc105730 con 0x7f7afc101b00 2023-08-25T02:36:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.936+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f7af000f450 con 0x7f7afc101b00 2023-08-25T02:36:33.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.936+0000 7f7af8ff9700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 0x7f7ae40a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:33.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.936+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f7af00cb6e0 con 0x7f7afc101b00 2023-08-25T02:36:33.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.936+0000 7f7afaffd700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 0x7f7ae40a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:33.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.937+0000 7f7afaffd700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 0x7f7ae40a2580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7afc10c240 tx=0x7f7aec00a380 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:33.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:33.941+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f7af0095100 con 0x7f7afc101b00 2023-08-25T02:36:33.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:33 smithi121 ceph-mon[100099]: pgmap v312: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:33 smithi121 ceph-mon[100099]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:33.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:33 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/419741349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:36:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:33 smithi122 ceph-mon[100457]: pgmap v312: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:33 smithi122 ceph-mon[100457]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:33.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:33 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/419741349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:36:34.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.131+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7afc197ea0 con 0x7f7afc101b00 2023-08-25T02:36:34.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.131+0000 7f7af8ff9700 1 -- 172.21.15.89:0/867576281 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7af0098720 con 0x7f7afc101b00 2023-08-25T02:36:34.133 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:36:34.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 msgr2=0x7f7ae40a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 0x7f7ae40a2580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7afc10c240 tx=0x7f7aec00a380 comp rx=0 tx=0).stop 2023-08-25T02:36:34.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 msgr2=0x7f7afc19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c9b0 tx=0x7f7af000ccc0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 shutdown_connections 2023-08-25T02:36:34.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f7ae40a00c0 0x7f7ae40a2580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7afc10aed0 0x7f7afc1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7afc101b00 0x7f7afc19c880 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 --2- 172.21.15.89:0/867576281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7afc102420 0x7f7afc19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 >> 172.21.15.89:0/867576281 conn(0x7f7afc0fb2c0 msgr2=0x7f7afc102870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:34.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 shutdown_connections 2023-08-25T02:36:34.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.134+0000 7f7b01f83700 1 -- 172.21.15.89:0/867576281 wait complete. 2023-08-25T02:36:34.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:36:34.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 -- 172.21.15.89:0/860876237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d00ff760 msgr2=0x7fb3d00ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 --2- 172.21.15.89:0/860876237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d00ff760 0x7fb3d00ffbe0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009990 tx=0x7fb3c4009ca0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 -- 172.21.15.89:0/860876237 shutdown_connections 2023-08-25T02:36:34.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 --2- 172.21.15.89:0/860876237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 0x7fb3d0105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 --2- 172.21.15.89:0/860876237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d0100120 0x7fb3d010c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 --2- 172.21.15.89:0/860876237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d00ff760 0x7fb3d00ffbe0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 -- 172.21.15.89:0/860876237 >> 172.21.15.89:0/860876237 conn(0x7fb3d00fb270 msgr2=0x7fb3d00fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:34.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 -- 172.21.15.89:0/860876237 shutdown_connections 2023-08-25T02:36:34.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.269+0000 7fb3d6574700 1 -- 172.21.15.89:0/860876237 wait complete. 2023-08-25T02:36:34.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 Processor -- start 2023-08-25T02:36:34.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 -- start start 2023-08-25T02:36:34.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00ff760 0x7fb3d019b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 0x7fb3d0195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0195cd0 con 0x7fb3d0100120 2023-08-25T02:36:34.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0195e40 con 0x7fb3d0105770 2023-08-25T02:36:34.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3d6574700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0196000 con 0x7fb3d00ff760 2023-08-25T02:36:34.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.270+0000 7fb3cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49462/0 (socket says 172.21.15.89:49462) 2023-08-25T02:36:34.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 -- 172.21.15.89:0/202621427 learned_addr learned my addr 172.21.15.89:0/202621427 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:34.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cffff700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00ff760 0x7fb3d019b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3d4b11700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 0x7fb3d0195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 -- 172.21.15.89:0/202621427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00ff760 msgr2=0x7fb3d019b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00ff760 0x7fb3d019b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 -- 172.21.15.89:0/202621427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 msgr2=0x7fb3d0195660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 0x7fb3d0195660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 -- 172.21.15.89:0/202621427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40095f0 con 0x7fb3d0100120 2023-08-25T02:36:34.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cf7fe700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c00 tx=0x7fb3c400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:34.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c401d070 con 0x7fb3d0100120 2023-08-25T02:36:34.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3c400fc20 con 0x7fb3d0100120 2023-08-25T02:36:34.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d01962e0 con 0x7fb3d0100120 2023-08-25T02:36:34.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4017650 con 0x7fb3d0100120 2023-08-25T02:36:34.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.271+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d01a25e0 con 0x7fb3d0100120 2023-08-25T02:36:34.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.272+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d0067af0 con 0x7fb3d0100120 2023-08-25T02:36:34.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.273+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fb3c4021410 con 0x7fb3d0100120 2023-08-25T02:36:34.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.274+0000 7fb3cd7fa700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 0x7fb3b80a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.274+0000 7fb3cffff700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 0x7fb3b80a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.274+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7fb3c40cd2c0 con 0x7fb3d0100120 2023-08-25T02:36:34.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.274+0000 7fb3cffff700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 0x7fb3b80a2580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0005950 tx=0x7fb3c00058e0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:34.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.278+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fb3c4096d70 con 0x7fb3d0100120 2023-08-25T02:36:34.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.469+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3d010f780 con 0x7fb3d0100120 2023-08-25T02:36:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.470+0000 7fb3cd7fa700 1 -- 172.21.15.89:0/202621427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3c409a390 con 0x7fb3d0100120 2023-08-25T02:36:34.471 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:36:34.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 msgr2=0x7fb3b80a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 0x7fb3b80a2580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0005950 tx=0x7fb3c00058e0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 msgr2=0x7fb3d019bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c00 tx=0x7fb3c400f510 comp rx=0 tx=0).stop 2023-08-25T02:36:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 shutdown_connections 2023-08-25T02:36:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fb3b80a00c0 0x7fb3b80a2580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3d0105770 0x7fb3d0195660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00ff760 0x7fb3d019b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 --2- 172.21.15.89:0/202621427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3d0100120 0x7fb3d019bb30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 >> 172.21.15.89:0/202621427 conn(0x7fb3d00fb270 msgr2=0x7fb3d00fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 shutdown_connections 2023-08-25T02:36:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.472+0000 7fb3d6574700 1 -- 172.21.15.89:0/202621427 wait complete. 2023-08-25T02:36:34.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-08-25T02:36:34.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 -- 172.21.15.89:0/2690224298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa84100e80 msgr2=0x7ffa84104420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 --2- 172.21.15.89:0/2690224298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa84100e80 0x7ffa84104420 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009990 tx=0x7ffa6c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 -- 172.21.15.89:0/2690224298 shutdown_connections 2023-08-25T02:36:34.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 --2- 172.21.15.89:0/2690224298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 0x7ffa841008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 --2- 172.21.15.89:0/2690224298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa8410c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 --2- 172.21.15.89:0/2690224298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa84100e80 0x7ffa84104420 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 -- 172.21.15.89:0/2690224298 >> 172.21.15.89:0/2690224298 conn(0x7ffa84074a00 msgr2=0x7ffa84074e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:34.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 -- 172.21.15.89:0/2690224298 shutdown_connections 2023-08-25T02:36:34.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.607+0000 7ffa8a041700 1 -- 172.21.15.89:0/2690224298 wait complete. 2023-08-25T02:36:34.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 Processor -- start 2023-08-25T02:36:34.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 -- start start 2023-08-25T02:36:34.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 0x7ffa84196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa84196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8419bd20 con 0x7ffa8419b330 2023-08-25T02:36:34.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8419be90 con 0x7ffa841004d0 2023-08-25T02:36:34.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa8a041700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8419c050 con 0x7ffa84104b00 2023-08-25T02:36:34.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49478/0 (socket says 172.21.15.89:49478) 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 -- 172.21.15.89:0/285867118 learned_addr learned my addr 172.21.15.89:0/285867118 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa837fe700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 0x7ffa84196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa82ffd700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa84196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 -- 172.21.15.89:0/285867118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 msgr2=0x7ffa84196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa84196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 -- 172.21.15.89:0/285867118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 msgr2=0x7ffa84196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:34.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 0x7ffa84196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:34.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 -- 172.21.15.89:0/285867118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c0095f0 con 0x7ffa8419b330 2023-08-25T02:36:34.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa82ffd700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa84196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:36:34.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.608+0000 7ffa83fff700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00ef30 tx=0x7ffa7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:34.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.609+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c018070 con 0x7ffa8419b330 2023-08-25T02:36:34.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.609+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa841a35e0 con 0x7ffa8419b330 2023-08-25T02:36:34.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.609+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa7c00f040 con 0x7ffa8419b330 2023-08-25T02:36:34.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.609+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c00cdb0 con 0x7ffa8419b330 2023-08-25T02:36:34.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.609+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa841a3a10 con 0x7ffa8419b330 2023-08-25T02:36:34.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.610+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa84066e80 con 0x7ffa8419b330 2023-08-25T02:36:34.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.611+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7ffa7c003a00 con 0x7ffa8419b330 2023-08-25T02:36:34.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.611+0000 7ffa80ff9700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 0x7ffa700a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:36:34.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.611+0000 7ffa837fe700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 0x7ffa700a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:36:34.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.611+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7ffa7c0cb210 con 0x7ffa8419b330 2023-08-25T02:36:34.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.612+0000 7ffa837fe700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 0x7ffa700a25c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009c60 tx=0x7ffa74009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:36:34.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.616+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7ffa7c094cc0 con 0x7ffa8419b330 2023-08-25T02:36:34.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:34 smithi089 ceph-mon[98732]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/4098694009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:36:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/867576281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:36:34.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:34 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/202621427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:36:34.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.863+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7ffa84101000 con 0x7ffa8419b330 2023-08-25T02:36:34.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.871+0000 7ffa80ff9700 1 -- 172.21.15.89:0/285867118 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v23) v1 ==== 81+0+728384 (secure 0 0 0) 0x7ffa7c0982e0 con 0x7ffa8419b330 2023-08-25T02:36:34.879 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:36:34.879 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:36:34.880 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 23, 2023-08-25T02:36:34.880 INFO:teuthology.orchestra.run.smithi089.stdout: "active_gid": 24889, 2023-08-25T02:36:34.880 INFO:teuthology.orchestra.run.smithi089.stdout: "active_name": "x", 2023-08-25T02:36:34.880 INFO:teuthology.orchestra.run.smithi089.stdout: "active_addrs": { 2023-08-25T02:36:34.881 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:34.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:34.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:34.881 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2023-08-25T02:36:34.881 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1258779165 2023-08-25T02:36:34.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:34.882 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2023-08-25T02:36:34.882 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2023-08-25T02:36:34.882 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1258779165 2023-08-25T02:36:34.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:34.883 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:34.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.883 INFO:teuthology.orchestra.run.smithi089.stdout: "active_addr": "172.21.15.89:6801/1258779165", 2023-08-25T02:36:34.883 INFO:teuthology.orchestra.run.smithi089.stdout: "active_change": "2023-08-25T02:26:30.360296+0000", 2023-08-25T02:36:34.884 INFO:teuthology.orchestra.run.smithi089.stdout: "active_mgr_features": 4540138322906710015, 2023-08-25T02:36:34.884 INFO:teuthology.orchestra.run.smithi089.stdout: "available": true, 2023-08-25T02:36:34.888 INFO:teuthology.orchestra.run.smithi089.stdout: "standbys": [ 2023-08-25T02:36:34.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:34.889 INFO:teuthology.orchestra.run.smithi089.stdout: "gid": 24890, 2023-08-25T02:36:34.889 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "y", 2023-08-25T02:36:34.890 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_features": 4540138322906710015, 2023-08-25T02:36:34.890 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2023-08-25T02:36:34.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:34.891 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2023-08-25T02:36:34.891 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:34.892 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:34.892 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:34.893 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:34.893 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:34.893 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:34.894 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.894 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:34.895 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.895 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.895 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.896 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2023-08-25T02:36:34.896 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.897 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.897 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.898 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:34.898 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:34.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.899 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.899 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.900 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.900 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.900 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.901 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:34.901 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:34.901 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:34.901 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:34.902 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:34.902 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:34.902 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:34.902 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:34.903 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.903 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.903 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.903 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.904 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:34.904 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:34.904 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.907 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:34.907 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:34.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.908 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.908 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:34.908 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.909 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.909 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:34.909 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:34.909 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:34.910 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:34.910 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:34.910 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:34.910 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:34.911 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:34.911 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.911 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.912 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:34.912 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:34.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.913 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.913 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.913 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:34.913 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.913 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.914 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.914 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.914 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.914 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.914 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.915 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2023-08-25T02:36:34.915 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2023-08-25T02:36:34.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.916 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.916 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.916 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.916 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.917 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.917 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2023-08-25T02:36:34.917 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.917 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.917 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.918 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2023-08-25T02:36:34.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2023-08-25T02:36:34.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.918 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.919 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2023-08-25T02:36:34.919 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.919 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.920 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.920 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2023-08-25T02:36:34.920 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.920 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.921 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2023-08-25T02:36:34.921 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2023-08-25T02:36:34.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.922 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.922 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.922 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.922 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.922 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.923 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.923 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2023-08-25T02:36:34.923 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.923 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.924 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2023-08-25T02:36:34.924 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2023-08-25T02:36:34.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.925 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.925 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.925 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.925 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.926 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.926 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2023-08-25T02:36:34.926 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.926 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.926 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.927 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2023-08-25T02:36:34.927 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2023-08-25T02:36:34.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:34.927 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.928 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2023-08-25T02:36:34.928 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.928 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.929 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.929 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2023-08-25T02:36:34.929 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.929 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2023-08-25T02:36:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2023-08-25T02:36:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.931 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.931 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.931 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.931 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.932 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.932 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2023-08-25T02:36:34.932 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.932 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.933 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.933 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2023-08-25T02:36:34.933 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2023-08-25T02:36:34.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.934 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.934 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:34.935 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.935 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.935 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.935 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2023-08-25T02:36:34.935 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.936 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.936 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.936 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2023-08-25T02:36:34.936 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2023-08-25T02:36:34.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.937 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.937 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.938 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.938 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.938 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.938 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2023-08-25T02:36:34.938 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.939 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.939 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:34.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:34.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:34.940 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2023-08-25T02:36:34.940 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:34.941 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:34.941 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:34.941 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2023-08-25T02:36:34.941 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2023-08-25T02:36:34.942 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.942 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.942 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.942 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:34.942 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.943 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.943 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.943 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2023-08-25T02:36:34.943 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.944 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.944 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2023-08-25T02:36:34.944 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2023-08-25T02:36:34.945 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.945 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.945 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2023-08-25T02:36:34.945 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.946 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.946 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.946 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2023-08-25T02:36:34.946 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:34.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.947 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.947 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2023-08-25T02:36:34.947 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2023-08-25T02:36:34.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:34.948 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.948 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.948 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:34.948 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:34.949 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:34.949 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.949 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-08-25T02:36:34.949 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:34.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.950 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.950 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2023-08-25T02:36:34.951 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2023-08-25T02:36:34.951 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:34.951 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.951 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.951 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2023-08-25T02:36:34.952 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:34.952 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2023-08-25T02:36:34.952 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.952 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-08-25T02:36:34.953 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.953 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2023-08-25T02:36:34.954 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2023-08-25T02:36:34.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.954 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.955 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2023-08-25T02:36:34.955 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.955 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.955 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-08-25T02:36:34.956 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-08-25T02:36:34.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.957 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2023-08-25T02:36:34.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2023-08-25T02:36:34.957 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:34.957 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.957 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.958 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2023-08-25T02:36:34.958 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2023-08-25T02:36:34.958 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2023-08-25T02:36:34.958 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.958 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2023-08-25T02:36:34.959 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-08-25T02:36:34.959 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.959 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.959 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2023-08-25T02:36:34.960 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2023-08-25T02:36:34.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.960 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.961 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2023-08-25T02:36:34.961 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.961 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.961 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.961 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2023-08-25T02:36:34.962 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:34.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.962 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.962 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2023-08-25T02:36:34.963 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2023-08-25T02:36:34.963 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:34.963 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.963 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.964 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:34.964 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:34.964 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:34.964 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.965 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-08-25T02:36:34.965 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:34.965 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.965 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.966 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:34.966 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:34.966 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.966 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.967 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.967 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.967 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.967 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.968 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:34.968 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:34.969 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:34.969 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:34.970 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:34.970 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:34.971 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:34.971 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:34.972 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.972 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.973 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.973 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.974 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:34.974 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:34.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.975 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.976 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.976 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:34.976 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.977 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.977 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.978 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.978 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.979 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.980 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.980 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:34.980 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:34.981 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.981 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.981 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.981 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:34.982 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.982 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.982 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:34.982 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:34.982 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:34.984 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.984 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.985 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.985 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.985 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:34.985 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:34.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:34.986 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.986 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:34.986 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.987 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.987 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.987 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:34.987 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.987 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.988 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.988 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2023-08-25T02:36:34.988 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2023-08-25T02:36:34.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:34.989 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.989 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.989 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:34.989 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.990 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.990 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.990 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-08-25T02:36:34.990 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.990 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.991 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.992 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.992 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2023-08-25T02:36:34.992 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.992 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.992 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:34.993 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2023-08-25T02:36:34.993 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2023-08-25T02:36:34.993 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap" 2023-08-25T02:36:34.993 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:34.993 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2023-08-25T02:36:34.994 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.994 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.994 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.994 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2023-08-25T02:36:34.995 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2023-08-25T02:36:34.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:34.995 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.995 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:34.996 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.996 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.996 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.996 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-08-25T02:36:34.996 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:34.997 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:34.997 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:34.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:34.997 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:34.997 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:34.998 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:34.999 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:34.999 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-08-25T02:36:34.999 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.000 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.000 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2023-08-25T02:36:35.000 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2023-08-25T02:36:35.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.001 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.001 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.001 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:35.001 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.002 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.003 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2023-08-25T02:36:35.003 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2023-08-25T02:36:35.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:35.003 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.004 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.004 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.004 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.004 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.005 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-08-25T02:36:35.005 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.005 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.005 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.005 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.006 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.006 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2023-08-25T02:36:35.006 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.007 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.007 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.007 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2023-08-25T02:36:35.007 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2023-08-25T02:36:35.007 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.008 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.008 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:35.008 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.008 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.009 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.009 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-08-25T02:36:35.009 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.010 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.010 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2023-08-25T02:36:35.010 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2023-08-25T02:36:35.010 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.011 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.011 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2023-08-25T02:36:35.011 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.011 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.012 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.012 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-08-25T02:36:35.012 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.012 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.012 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.013 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2023-08-25T02:36:35.013 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2023-08-25T02:36:35.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.013 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.014 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.014 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2023-08-25T02:36:35.014 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.014 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.014 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.015 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-08-25T02:36:35.015 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.015 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.016 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2023-08-25T02:36:35.016 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2023-08-25T02:36:35.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.016 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.017 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.017 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.017 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.017 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.017 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-08-25T02:36:35.018 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-08-25T02:36:35.018 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.018 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.018 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2023-08-25T02:36:35.019 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2023-08-25T02:36:35.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.019 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.019 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:35.020 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.020 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.020 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.020 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2023-08-25T02:36:35.021 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.021 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.022 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2023-08-25T02:36:35.023 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.023 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.023 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.023 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-08-25T02:36:35.023 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.024 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.024 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.024 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2023-08-25T02:36:35.024 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2023-08-25T02:36:35.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.025 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.025 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.025 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.026 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.026 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.026 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-08-25T02:36:35.026 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.026 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.027 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.028 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.028 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.028 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.029 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.029 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-08-25T02:36:35.029 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.029 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.029 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.030 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2023-08-25T02:36:35.030 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2023-08-25T02:36:35.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.031 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.031 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.031 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.031 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-08-25T02:36:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.033 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2023-08-25T02:36:35.033 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2023-08-25T02:36:35.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.033 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-08-25T02:36:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.035 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:35.035 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.035 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.035 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.036 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2023-08-25T02:36:35.036 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2023-08-25T02:36:35.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.036 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.037 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2023-08-25T02:36:35.037 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.037 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.038 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.038 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2023-08-25T02:36:35.039 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.039 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.040 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.041 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2023-08-25T02:36:35.041 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2023-08-25T02:36:35.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.042 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.042 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.042 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-08-25T02:36:35.042 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.042 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.043 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.043 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2023-08-25T02:36:35.043 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.043 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.044 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2023-08-25T02:36:35.044 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2023-08-25T02:36:35.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.044 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-08-25T02:36:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.046 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2023-08-25T02:36:35.047 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2023-08-25T02:36:35.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.047 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.047 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-08-25T02:36:35.048 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.048 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.048 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.048 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2023-08-25T02:36:35.048 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2023-08-25T02:36:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2023-08-25T02:36:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-08-25T02:36:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2023-08-25T02:36:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2023-08-25T02:36:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2023-08-25T02:36:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-08-25T02:36:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2023-08-25T02:36:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2023-08-25T02:36:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-08-25T02:36:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2023-08-25T02:36:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-08-25T02:36:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2023-08-25T02:36:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2023-08-25T02:36:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2023-08-25T02:36:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-08-25T02:36:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2023-08-25T02:36:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2023-08-25T02:36:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2023-08-25T02:36:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-08-25T02:36:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2023-08-25T02:36:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2023-08-25T02:36:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2023-08-25T02:36:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2023-08-25T02:36:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2023-08-25T02:36:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-08-25T02:36:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2023-08-25T02:36:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2023-08-25T02:36:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2023-08-25T02:36:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2023-08-25T02:36:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2023-08-25T02:36:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-08-25T02:36:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2023-08-25T02:36:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2023-08-25T02:36:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2023-08-25T02:36:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2023-08-25T02:36:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2023-08-25T02:36:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2023-08-25T02:36:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-08-25T02:36:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2023-08-25T02:36:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2023-08-25T02:36:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2023-08-25T02:36:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-08-25T02:36:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2023-08-25T02:36:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-08-25T02:36:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2023-08-25T02:36:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2023-08-25T02:36:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2023-08-25T02:36:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2023-08-25T02:36:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2023-08-25T02:36:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2023-08-25T02:36:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2023-08-25T02:36:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2023-08-25T02:36:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2023-08-25T02:36:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2023-08-25T02:36:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2023-08-25T02:36:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2023-08-25T02:36:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2023-08-25T02:36:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2023-08-25T02:36:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-08-25T02:36:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-08-25T02:36:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2023-08-25T02:36:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-08-25T02:36:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-08-25T02:36:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-08-25T02:36:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-08-25T02:36:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2023-08-25T02:36:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-08-25T02:36:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2023-08-25T02:36:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2023-08-25T02:36:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2023-08-25T02:36:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2023-08-25T02:36:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2023-08-25T02:36:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-08-25T02:36:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2023-08-25T02:36:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2023-08-25T02:36:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2023-08-25T02:36:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2023-08-25T02:36:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2023-08-25T02:36:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-08-25T02:36:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-08-25T02:36:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2023-08-25T02:36:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2023-08-25T02:36:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-08-25T02:36:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2023-08-25T02:36:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2023-08-25T02:36:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-08-25T02:36:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2023-08-25T02:36:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2023-08-25T02:36:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-08-25T02:36:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2023-08-25T02:36:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2023-08-25T02:36:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.156 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.156 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-08-25T02:36:35.156 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.156 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.157 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.158 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.158 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.158 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.159 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.159 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-08-25T02:36:35.159 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.159 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.160 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.160 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2023-08-25T02:36:35.160 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2023-08-25T02:36:35.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.161 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.161 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2023-08-25T02:36:35.161 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.161 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.163 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2023-08-25T02:36:35.163 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2023-08-25T02:36:35.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.163 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.164 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.164 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.164 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.164 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.165 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-08-25T02:36:35.165 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.165 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.165 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.166 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2023-08-25T02:36:35.166 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2023-08-25T02:36:35.166 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.166 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.166 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.167 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.167 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.167 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.167 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.168 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-08-25T02:36:35.168 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.168 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.169 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2023-08-25T02:36:35.169 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2023-08-25T02:36:35.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.169 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.170 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.170 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.171 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.171 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.171 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-08-25T02:36:35.171 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.171 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.172 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.172 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2023-08-25T02:36:35.172 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.173 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2023-08-25T02:36:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.175 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2023-08-25T02:36:35.175 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2023-08-25T02:36:35.176 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.176 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.176 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.177 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.177 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.177 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.177 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-08-25T02:36:35.178 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.178 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.178 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.179 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2023-08-25T02:36:35.179 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2023-08-25T02:36:35.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.179 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.180 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.180 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.180 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.180 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.181 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.181 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-08-25T02:36:35.181 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.181 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.181 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.182 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2023-08-25T02:36:35.183 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.183 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.183 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.183 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.183 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.184 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.184 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.184 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.184 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.185 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.185 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.185 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.185 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.185 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.186 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.186 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.186 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.186 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.186 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.187 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.187 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.188 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.188 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.188 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.189 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.189 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.189 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.190 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.190 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.190 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.190 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.191 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.191 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.192 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.192 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.192 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.193 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.193 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.193 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.193 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.193 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.194 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.194 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.194 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.194 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.194 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.195 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.195 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.195 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.195 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.196 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.196 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.196 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.197 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.197 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.197 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.197 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.197 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.198 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.198 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.198 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.199 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2023-08-25T02:36:35.199 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2023-08-25T02:36:35.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.199 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.200 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2023-08-25T02:36:35.200 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.200 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.200 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.200 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2023-08-25T02:36:35.201 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.201 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.201 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.201 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2023-08-25T02:36:35.202 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2023-08-25T02:36:35.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.202 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.202 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2023-08-25T02:36:35.203 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.203 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.203 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.203 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2023-08-25T02:36:35.203 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.204 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2023-08-25T02:36:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.207 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.207 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.207 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.207 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.207 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.208 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.208 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.209 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2023-08-25T02:36:35.209 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2023-08-25T02:36:35.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.209 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.210 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.210 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.210 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.210 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.211 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.211 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.211 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.211 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.212 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-08-25T02:36:35.212 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-08-25T02:36:35.212 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.212 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.213 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.213 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.213 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.213 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.213 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.214 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.214 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.214 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.214 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2023-08-25T02:36:35.215 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2023-08-25T02:36:35.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.215 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.216 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.217 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.217 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.217 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.218 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.218 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.219 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.219 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-08-25T02:36:35.219 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-08-25T02:36:35.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.220 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.220 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.220 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.221 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.221 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.221 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.221 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.222 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.222 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2023-08-25T02:36:35.222 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2023-08-25T02:36:35.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.223 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.223 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.223 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.225 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.225 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-08-25T02:36:35.225 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-08-25T02:36:35.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.226 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.227 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.227 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.227 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.227 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.228 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-08-25T02:36:35.228 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-08-25T02:36:35.228 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.229 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.230 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.230 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.230 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.231 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.231 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.232 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.232 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.233 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-08-25T02:36:35.234 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-08-25T02:36:35.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.234 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.234 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.235 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.235 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.235 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.235 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.235 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.236 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.236 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.236 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.236 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-08-25T02:36:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-08-25T02:36:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.238 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.238 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.238 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.238 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.238 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.239 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.239 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.239 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2023-08-25T02:36:35.239 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-08-25T02:36:35.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.240 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.240 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.241 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.241 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.241 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.241 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.241 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.242 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.242 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2023-08-25T02:36:35.242 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-08-25T02:36:35.243 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.243 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.243 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.243 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.244 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.244 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.244 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.244 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.244 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.245 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.245 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-25T02:36:35.245 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-08-25T02:36:35.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.246 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.246 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.246 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.246 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.247 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.247 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.247 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.247 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.247 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.248 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-08-25T02:36:35.248 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-08-25T02:36:35.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.248 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.249 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.249 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.249 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.249 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.249 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.250 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.250 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.250 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.251 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2023-08-25T02:36:35.251 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2023-08-25T02:36:35.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.251 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.252 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:35.252 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.252 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2023-08-25T02:36:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2023-08-25T02:36:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2023-08-25T02:36:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2023-08-25T02:36:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-08-25T02:36:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-08-25T02:36:35.264 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.264 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.264 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.264 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-08-25T02:36:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-08-25T02:36:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-08-25T02:36:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2023-08-25T02:36:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-08-25T02:36:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.273 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.273 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.273 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.273 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.274 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.274 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.274 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.275 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.275 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.276 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2023-08-25T02:36:35.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2023-08-25T02:36:35.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.277 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.278 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.278 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.278 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.279 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.279 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.279 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.279 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.279 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-08-25T02:36:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-08-25T02:36:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.280 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.281 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.282 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-08-25T02:36:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-08-25T02:36:35.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.284 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.284 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.284 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.284 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.285 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.285 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.285 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.285 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.285 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.286 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-08-25T02:36:35.286 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-08-25T02:36:35.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.286 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.287 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.288 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.288 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.289 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-08-25T02:36:35.289 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-08-25T02:36:35.289 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.289 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.289 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.290 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.291 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.291 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-08-25T02:36:35.291 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-08-25T02:36:35.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.292 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.292 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.292 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.293 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.293 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.293 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.293 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.293 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-08-25T02:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-08-25T02:36:35.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.295 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.295 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.295 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.296 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.296 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.296 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.296 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.296 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.297 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.297 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-08-25T02:36:35.297 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-08-25T02:36:35.297 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.300 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-08-25T02:36:35.300 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-08-25T02:36:35.300 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.300 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.301 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.301 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.301 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.301 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.301 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.302 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.302 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.303 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2023-08-25T02:36:35.303 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2023-08-25T02:36:35.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.303 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.303 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.304 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.304 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.304 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.304 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.304 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.305 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.305 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.305 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-08-25T02:36:35.306 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-08-25T02:36:35.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.306 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.306 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-08-25T02:36:35.307 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.307 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.307 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.307 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.307 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.308 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.308 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-08-25T02:36:35.308 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-08-25T02:36:35.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.309 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.309 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.309 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.310 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.311 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-08-25T02:36:35.311 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-08-25T02:36:35.311 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.312 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.312 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2023-08-25T02:36:35.312 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.312 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.313 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.313 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.313 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.314 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.314 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.314 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2023-08-25T02:36:35.314 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.315 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.316 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.316 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.316 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.316 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.316 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.317 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2023-08-25T02:36:35.317 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2023-08-25T02:36:35.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.318 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.319 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.319 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.319 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.320 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.320 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.320 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.320 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.320 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.321 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-08-25T02:36:35.321 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-08-25T02:36:35.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.321 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.322 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.322 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:35.322 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.322 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.322 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.323 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.323 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.323 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.323 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.324 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2023-08-25T02:36:35.324 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2023-08-25T02:36:35.324 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.325 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.325 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.325 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.325 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.326 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.326 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.326 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.326 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.326 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.327 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2023-08-25T02:36:35.327 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2023-08-25T02:36:35.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.327 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.328 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.328 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.328 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.328 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.330 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-08-25T02:36:35.330 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-08-25T02:36:35.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.330 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.331 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.332 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.332 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.332 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-08-25T02:36:35.333 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-08-25T02:36:35.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.333 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.333 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.334 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.334 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.334 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.334 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.335 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.335 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.335 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.335 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-08-25T02:36:35.336 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-08-25T02:36:35.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.336 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.336 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:35.337 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.337 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.337 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.337 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.337 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.338 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.338 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2023-08-25T02:36:35.338 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.339 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.340 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.340 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.340 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.341 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.342 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.342 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.342 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.342 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.344 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2023-08-25T02:36:35.344 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2023-08-25T02:36:35.344 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.344 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.345 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.345 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.345 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.345 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.345 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.346 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2023-08-25T02:36:35.346 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.346 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.346 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.347 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2023-08-25T02:36:35.347 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2023-08-25T02:36:35.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.347 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.348 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2023-08-25T02:36:35.348 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.348 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.348 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.349 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.349 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.349 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.350 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:35.350 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:35.350 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.350 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.350 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.351 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.352 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.352 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.352 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.353 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.353 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.353 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.353 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.354 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.354 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.354 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.354 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.354 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.355 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.355 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.355 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.355 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.356 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.356 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.356 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.357 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.357 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.357 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.358 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.358 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.358 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.358 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.358 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.359 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.359 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.359 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.359 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.360 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.360 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.360 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.360 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.360 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.362 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.362 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.362 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.362 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.362 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.363 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.363 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.363 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.363 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.364 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.364 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.364 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.365 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.365 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.365 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.365 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2023-08-25T02:36:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2023-08-25T02:36:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.369 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2023-08-25T02:36:35.369 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.369 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.369 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.370 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.372 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.372 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.373 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:35.373 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:35.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.373 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.374 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.374 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:35.374 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.374 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.375 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.375 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.375 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.375 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.375 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.376 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:35.377 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:35.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.377 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.378 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2023-08-25T02:36:35.378 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.378 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.378 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.378 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.379 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.379 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.379 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.379 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:35.380 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:35.380 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.380 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.381 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.381 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.381 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.381 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.382 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.382 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.382 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.383 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2023-08-25T02:36:35.383 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2023-08-25T02:36:35.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.384 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2023-08-25T02:36:35.385 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.385 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.385 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.385 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.385 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:35.387 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.387 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.387 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.387 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2023-08-25T02:36:35.388 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.388 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.388 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.388 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.389 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2023-08-25T02:36:35.389 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.389 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.390 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.390 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.390 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.390 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:35.391 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:35.391 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.391 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.391 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.391 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:35.392 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:35.392 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:35.392 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.392 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.392 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2023-08-25T02:36:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2023-08-25T02:36:35.394 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.394 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.394 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.394 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.394 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.395 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.395 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.395 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.395 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.395 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.396 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.397 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2023-08-25T02:36:35.397 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.397 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.397 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.397 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2023-08-25T02:36:35.398 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2023-08-25T02:36:35.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.398 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.398 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.399 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.400 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.400 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.400 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.401 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.402 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.402 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.402 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.403 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.403 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.403 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.404 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.404 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.405 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.405 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.405 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.406 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.406 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.407 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.407 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.407 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.407 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.408 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.408 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.408 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.408 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.408 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.409 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.409 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.409 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.410 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.410 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.410 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.412 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.412 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.412 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.412 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.412 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.413 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.413 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.413 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.413 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.414 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.414 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.414 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.415 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.415 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.415 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.415 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.415 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.416 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2023-08-25T02:36:35.417 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2023-08-25T02:36:35.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.417 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.417 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2023-08-25T02:36:35.418 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.418 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.418 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.418 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-08-25T02:36:35.418 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2023-08-25T02:36:35.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.420 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.420 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2023-08-25T02:36:35.420 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.421 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.422 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2023-08-25T02:36:35.422 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2023-08-25T02:36:35.422 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.422 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.423 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.423 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2023-08-25T02:36:35.423 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.423 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.424 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.424 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2023-08-25T02:36:35.424 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.424 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.425 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2023-08-25T02:36:35.425 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2023-08-25T02:36:35.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.425 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.426 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.426 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:35.426 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.426 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.426 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.427 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2023-08-25T02:36:35.428 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2023-08-25T02:36:35.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.428 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-08-25T02:36:35.429 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.430 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.430 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.430 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:35.430 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:35.431 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.431 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.431 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.431 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:35.431 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.432 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.432 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.432 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2023-08-25T02:36:35.432 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.432 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.433 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.433 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2023-08-25T02:36:35.433 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2023-08-25T02:36:35.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.434 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-08-25T02:36:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2023-08-25T02:36:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.439 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.440 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.440 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.440 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.441 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.441 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.441 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.441 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.442 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.442 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.442 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.442 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.442 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.443 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.443 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.443 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.443 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.443 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.444 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.444 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.444 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.444 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.446 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.446 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.446 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.447 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.447 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.447 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.447 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.448 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.448 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.448 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.448 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.448 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.449 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.449 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.450 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.450 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.450 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.451 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.451 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.451 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.451 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.452 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.452 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.452 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.452 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.453 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.453 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2023-08-25T02:36:35.453 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2023-08-25T02:36:35.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.454 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.454 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:35.454 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.454 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.456 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2023-08-25T02:36:35.456 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2023-08-25T02:36:35.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.456 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.456 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.457 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2023-08-25T02:36:35.457 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.457 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.457 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.457 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.458 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.458 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.458 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:35.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:35.459 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.459 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.459 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.459 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:35.460 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.460 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.460 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.460 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.460 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.461 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.462 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2023-08-25T02:36:35.463 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2023-08-25T02:36:35.463 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.463 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.463 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.463 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2023-08-25T02:36:35.464 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.464 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.464 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.464 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-08-25T02:36:35.464 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.465 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.465 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2023-08-25T02:36:35.465 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2023-08-25T02:36:35.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.466 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.466 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2023-08-25T02:36:35.466 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.467 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.467 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.467 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-08-25T02:36:35.467 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.467 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.468 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.468 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2023-08-25T02:36:35.468 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2023-08-25T02:36:35.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.469 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.469 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.469 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.469 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.469 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.470 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.470 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2023-08-25T02:36:35.470 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.470 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.470 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.471 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:35.471 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:35.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.472 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.472 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2023-08-25T02:36:35.472 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2023-08-25T02:36:35.473 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.473 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.473 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-08-25T02:36:35.473 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.473 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.474 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.474 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.474 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.476 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.476 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.476 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.476 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.477 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.477 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.477 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.478 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.478 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.478 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.478 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.479 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.479 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.479 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.479 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.480 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.480 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.480 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.480 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.480 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.481 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.482 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.482 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.482 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.482 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2023-08-25T02:36:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2023-08-25T02:36:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2023-08-25T02:36:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2023-08-25T02:36:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2023-08-25T02:36:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2023-08-25T02:36:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2023-08-25T02:36:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2023-08-25T02:36:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2023-08-25T02:36:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2023-08-25T02:36:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2023-08-25T02:36:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2023-08-25T02:36:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2023-08-25T02:36:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2023-08-25T02:36:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2023-08-25T02:36:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2023-08-25T02:36:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2023-08-25T02:36:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2023-08-25T02:36:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2023-08-25T02:36:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2023-08-25T02:36:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2023-08-25T02:36:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2023-08-25T02:36:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2023-08-25T02:36:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2023-08-25T02:36:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2023-08-25T02:36:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-08-25T02:36:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2023-08-25T02:36:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2023-08-25T02:36:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2023-08-25T02:36:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-08-25T02:36:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2023-08-25T02:36:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2023-08-25T02:36:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2023-08-25T02:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-08-25T02:36:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2023-08-25T02:36:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2023-08-25T02:36:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2023-08-25T02:36:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2023-08-25T02:36:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.579 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.579 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2023-08-25T02:36:35.579 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2023-08-25T02:36:35.579 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.580 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.580 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.580 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2023-08-25T02:36:35.580 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.580 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.582 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2023-08-25T02:36:35.582 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2023-08-25T02:36:35.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.582 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.583 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.583 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.583 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.583 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.583 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2023-08-25T02:36:35.584 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.584 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.584 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.584 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2023-08-25T02:36:35.585 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2023-08-25T02:36:35.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.585 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.585 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-08-25T02:36:35.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.587 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.588 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2023-08-25T02:36:35.588 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.588 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.588 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.588 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.589 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.589 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.589 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.590 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.590 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.590 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.590 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.590 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.591 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.592 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.592 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.592 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.593 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.593 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.593 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.594 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.594 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.594 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.594 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.594 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.595 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.595 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.595 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.596 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.597 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.597 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.597 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.597 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.598 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.598 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.598 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.598 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.598 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.599 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.599 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.599 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.600 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.600 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.600 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.601 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.601 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.601 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.601 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.601 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.602 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.603 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2023-08-25T02:36:35.603 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.603 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.603 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.604 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.604 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.604 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.604 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.605 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.605 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.605 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.606 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.606 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.606 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.607 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.607 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.607 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.607 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.607 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.608 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.608 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.608 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.609 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.609 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.609 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.610 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.610 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.610 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.610 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.611 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.611 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.611 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.612 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.612 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.613 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.613 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.613 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.613 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.614 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.614 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.614 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.614 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.614 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.615 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.615 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.615 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.615 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.615 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.616 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.616 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.617 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.617 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.618 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.618 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.618 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.618 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.619 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.619 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.619 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.620 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2023-08-25T02:36:35.620 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.621 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.621 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.621 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.621 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.622 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.622 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.622 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.622 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.623 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.623 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.623 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.623 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.623 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.624 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.624 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.624 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.624 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.624 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.625 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.625 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.625 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.625 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.626 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.626 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.626 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.627 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.627 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.627 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.627 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.628 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.628 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.629 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.629 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.629 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.630 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.630 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.630 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.630 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.631 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.631 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.631 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.631 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.631 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.632 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.632 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.632 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.632 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.633 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.633 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.633 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.633 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.633 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.634 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.634 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.634 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.634 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.634 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.635 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.635 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.635 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.636 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:35.636 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.636 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.637 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.637 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2023-08-25T02:36:35.637 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2023-08-25T02:36:35.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.637 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.638 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.638 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.638 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.638 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-08-25T02:36:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2023-08-25T02:36:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2023-08-25T02:36:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2023-08-25T02:36:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2023-08-25T02:36:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2023-08-25T02:36:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2023-08-25T02:36:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.671 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.671 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.671 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.671 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.671 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2023-08-25T02:36:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2023-08-25T02:36:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "noautoscale": { 2023-08-25T02:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "noautoscale", 2023-08-25T02:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "global autoscale flag", 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2023-08-25T02:36:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2023-08-25T02:36:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2023-08-25T02:36:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2023-08-25T02:36:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-08-25T02:36:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2023-08-25T02:36:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2023-08-25T02:36:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.726 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.726 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.726 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.727 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.727 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.728 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.728 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.728 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.728 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.730 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.732 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.732 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.732 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.732 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.732 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.733 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.733 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.733 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.733 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.734 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.734 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.734 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.734 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.734 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.736 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2023-08-25T02:36:35.736 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2023-08-25T02:36:35.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.736 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.737 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2023-08-25T02:36:35.737 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.737 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.737 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.738 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2023-08-25T02:36:35.738 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.738 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.739 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:35.739 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:35.739 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.739 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.739 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.740 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:35.740 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.740 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.740 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.740 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2023-08-25T02:36:35.741 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.741 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.741 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.742 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2023-08-25T02:36:35.742 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.742 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.743 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.743 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2023-08-25T02:36:35.743 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2023-08-25T02:36:35.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.743 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.744 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.744 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.744 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.744 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.744 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.745 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.745 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.745 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.745 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.746 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.747 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.747 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.747 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.747 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-08-25T02:36:35.747 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-08-25T02:36:35.748 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.748 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.748 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.748 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.749 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.749 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.749 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.749 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.749 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.750 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.750 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.750 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.750 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.750 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.751 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.752 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.752 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.752 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.753 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.753 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.753 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.753 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.753 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.754 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.754 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.754 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.754 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.754 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.755 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.756 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.756 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.756 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.757 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.757 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.757 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.757 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.758 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.758 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.758 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.758 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.758 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.759 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.759 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.759 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.759 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.760 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.760 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.760 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.760 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.761 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.761 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.761 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.761 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.761 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2023-08-25T02:36:35.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.763 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.763 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.763 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.763 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.764 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.764 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.764 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.764 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.765 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.765 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2023-08-25T02:36:35.765 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-08-25T02:36:35.765 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.766 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.766 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2023-08-25T02:36:35.766 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.766 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.767 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.767 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.767 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.767 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.767 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.768 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2023-08-25T02:36:35.768 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2023-08-25T02:36:35.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.768 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.769 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.769 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2023-08-25T02:36:35.769 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.769 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.769 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.770 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.770 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.770 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.770 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.771 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:35.771 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:35.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.771 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.772 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:35.772 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.772 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.772 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.773 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-08-25T02:36:35.773 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.773 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.774 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:35.774 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:35.774 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.774 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.774 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.775 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2023-08-25T02:36:35.775 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.775 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.775 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.775 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-08-25T02:36:35.776 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.776 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.776 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2023-08-25T02:36:35.777 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2023-08-25T02:36:35.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.777 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.777 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2023-08-25T02:36:35.778 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.778 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.778 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.778 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.778 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.779 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.779 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.779 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:35.779 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:35.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.780 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.780 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2023-08-25T02:36:35.780 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.781 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.782 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.782 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.782 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:35.783 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:35.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.783 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.783 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:35.784 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:35.784 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:35.784 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.784 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.784 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.785 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.785 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.786 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.787 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.787 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.787 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.787 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.787 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.789 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.789 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.789 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.789 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.789 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.790 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.790 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.790 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.790 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.791 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.791 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.791 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.792 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.793 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.793 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.793 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.794 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.794 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.794 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.795 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.796 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.796 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.796 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.796 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.797 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.797 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.797 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.797 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.798 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.798 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.798 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.798 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.799 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.799 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.799 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.799 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.799 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.800 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.800 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.800 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.800 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2023-08-25T02:36:35.801 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2023-08-25T02:36:35.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.801 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.802 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.803 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.803 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.803 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.803 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2023-08-25T02:36:35.804 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2023-08-25T02:36:35.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.804 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.804 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.805 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.806 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.806 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.806 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2023-08-25T02:36:35.806 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2023-08-25T02:36:35.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.807 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.807 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.807 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.808 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.808 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.808 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.808 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.809 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2023-08-25T02:36:35.810 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.810 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.810 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.810 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2023-08-25T02:36:35.810 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2023-08-25T02:36:35.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.811 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.811 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.811 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.812 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.812 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.812 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.812 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.813 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.813 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:35.813 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:35.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.814 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.815 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.815 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.815 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.815 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.817 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.817 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.817 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.817 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.817 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.818 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.818 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.818 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.818 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.818 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.819 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.819 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.819 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.819 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.820 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.820 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.820 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.821 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.822 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.822 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.822 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.822 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.823 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.823 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.823 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.824 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.824 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.824 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.824 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.824 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.825 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.825 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.825 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.825 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.825 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.826 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.826 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.826 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.826 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.828 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.828 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.828 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.828 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.828 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.829 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.829 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.829 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.829 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:35.830 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:35.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.830 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.830 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.831 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.831 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.831 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.831 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.831 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.832 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.832 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:35.833 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:35.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.833 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.834 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.834 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.834 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2023-08-25T02:36:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.837 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.837 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.837 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.838 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.838 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.838 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.838 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.839 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.839 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.839 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.839 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.839 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.840 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.840 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.840 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.840 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.841 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.841 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.841 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.842 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.842 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.842 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.842 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.843 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.843 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.843 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.844 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.844 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.844 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.845 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.845 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.845 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.845 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.845 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.846 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.846 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.846 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.846 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.846 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.849 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.849 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.849 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.849 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.849 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.850 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.850 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.851 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2023-08-25T02:36:35.851 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.851 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.852 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.852 INFO:teuthology.orchestra.run.smithi089.stdout: "drive_group_interval": { 2023-08-25T02:36:35.852 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "drive_group_interval", 2023-08-25T02:36:35.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.852 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300.0", 2023-08-25T02:36:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.854 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-08-25T02:36:35.854 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.854 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.855 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.856 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.856 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.856 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.856 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.856 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.857 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.857 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.857 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.857 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.857 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.858 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.858 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.858 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.859 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.859 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.859 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.859 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.859 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.860 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.861 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.861 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.861 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.861 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.862 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.862 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.862 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.862 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.863 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.863 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.863 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.863 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.863 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.864 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.864 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.864 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.864 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.864 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.865 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.865 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.865 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.865 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.866 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.866 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.866 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.866 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.866 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.868 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.868 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2023-08-25T02:36:35.868 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2023-08-25T02:36:35.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.869 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.869 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2023-08-25T02:36:35.869 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.869 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.870 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.870 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-08-25T02:36:35.870 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.870 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.870 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.871 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2023-08-25T02:36:35.872 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.872 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.872 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.872 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.872 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.873 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.873 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.873 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.874 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.874 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.874 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.874 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.874 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.875 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.876 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.876 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.876 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.876 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.877 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.877 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.877 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.878 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.878 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.878 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.878 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.878 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.879 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.880 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.880 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.880 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.881 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.882 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.882 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.882 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.882 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.882 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.883 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.883 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.883 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.883 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.884 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.884 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.884 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.884 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.884 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.885 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.885 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.885 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.885 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.886 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2023-08-25T02:36:35.886 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2023-08-25T02:36:35.886 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.886 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.887 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.887 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.887 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.887 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.887 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.888 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.888 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.888 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.888 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2023-08-25T02:36:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2023-08-25T02:36:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.890 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.891 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.891 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2023-08-25T02:36:35.891 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2023-08-25T02:36:35.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.892 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.892 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.892 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.893 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.893 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.893 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.893 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.893 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.894 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.894 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2023-08-25T02:36:35.894 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2023-08-25T02:36:35.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.895 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.895 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.895 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.895 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.895 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.896 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.896 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.896 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.897 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.897 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2023-08-25T02:36:35.897 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2023-08-25T02:36:35.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:35.898 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.898 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.898 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.899 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.899 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.899 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.899 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.899 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.900 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.900 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2023-08-25T02:36:35.900 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2023-08-25T02:36:35.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.901 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.901 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.901 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.902 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.903 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2023-08-25T02:36:35.903 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2023-08-25T02:36:35.903 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.903 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.905 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.905 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.905 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.906 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2023-08-25T02:36:35.906 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2023-08-25T02:36:35.906 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.906 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.907 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.908 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.908 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.908 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2023-08-25T02:36:35.908 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2023-08-25T02:36:35.909 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:35.909 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.909 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.909 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.909 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:35.910 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2023-08-25T02:36:35.910 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.910 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.910 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.911 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2023-08-25T02:36:35.911 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2023-08-25T02:36:35.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.912 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.912 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.913 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.913 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.913 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.913 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.913 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2023-08-25T02:36:35.914 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.915 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.915 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.915 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.915 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.916 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.916 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.916 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.916 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.916 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.917 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2023-08-25T02:36:35.917 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2023-08-25T02:36:35.917 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.918 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.918 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.918 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.918 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.918 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.919 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.919 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.919 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.919 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.919 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2023-08-25T02:36:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2023-08-25T02:36:35.922 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2023-08-25T02:36:35.922 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.922 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.922 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.922 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.923 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.923 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.923 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.923 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2023-08-25T02:36:35.923 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.924 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.924 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2023-08-25T02:36:35.924 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2023-08-25T02:36:35.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.925 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.925 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.925 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.926 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.926 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.926 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2023-08-25T02:36:35.926 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.926 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.927 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.928 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.928 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.928 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.928 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.929 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.930 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.930 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.930 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.930 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.931 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.931 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.931 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.932 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.932 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.932 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.933 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.933 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.933 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.933 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.934 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.934 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.934 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.935 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.935 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.935 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.936 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.936 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.936 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.936 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.938 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.939 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.939 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.939 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.940 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.940 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.940 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.940 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.940 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.941 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.941 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.941 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.941 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2023-08-25T02:36:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.943 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.944 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.944 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.944 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.944 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.944 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.945 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.945 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.945 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.945 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.945 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.946 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.946 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.946 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.946 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.947 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.947 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.947 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.947 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.948 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.948 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.948 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.948 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.948 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.949 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.949 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.949 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.949 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.950 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.950 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.950 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.951 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.951 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.951 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.951 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.951 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.953 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.954 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.955 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.955 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.955 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.955 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2023-08-25T02:36:35.957 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.957 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.957 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.958 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.959 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.959 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.959 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.959 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.959 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.960 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.960 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.960 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.960 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.960 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.961 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.961 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.961 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.963 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.964 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.964 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.964 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.964 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.965 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.965 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.965 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.966 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.966 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.966 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.966 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.966 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.967 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.967 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.967 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.967 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.967 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.968 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.968 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.968 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.968 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.970 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.970 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.970 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.970 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.970 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.971 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.972 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2023-08-25T02:36:35.972 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.972 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.972 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.972 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-08-25T02:36:35.973 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.974 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.974 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.974 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.974 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.975 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.975 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:35.975 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:35.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2023-08-25T02:36:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.977 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.977 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.977 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.977 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.977 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.978 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:35.978 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:35.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.978 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.979 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.979 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:35.979 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.979 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.979 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.980 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.980 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.980 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.980 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.980 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.982 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:35.982 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:35.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.982 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.983 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.983 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.983 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.983 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.983 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.984 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.984 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.984 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.984 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.985 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:35.986 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.986 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.986 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:35.986 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:35.986 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:35.987 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:35.987 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:35.987 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:35.987 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:35.988 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:35.988 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.988 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.988 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.988 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.989 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:35.989 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:35.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.989 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:35.990 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:35.990 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.990 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.990 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:35.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:35.992 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2023-08-25T02:36:35.992 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:35.993 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:35.993 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:35.993 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2023-08-25T02:36:35.993 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2023-08-25T02:36:35.993 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.994 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.994 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.994 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.994 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.994 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.995 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.995 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2023-08-25T02:36:35.995 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.996 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.996 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2023-08-25T02:36:35.996 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2023-08-25T02:36:35.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:35.997 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:35.997 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:35.997 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:35.997 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:35.998 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:35.998 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:35.998 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-08-25T02:36:35.998 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:35.998 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:35.999 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:35.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:35.999 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2023-08-25T02:36:35.999 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2023-08-25T02:36:35.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.000 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.000 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.000 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.000 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.000 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.001 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.001 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-08-25T02:36:36.001 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.001 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.001 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.002 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2023-08-25T02:36:36.002 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2023-08-25T02:36:36.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.002 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.004 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-08-25T02:36:36.004 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.004 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.004 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.005 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2023-08-25T02:36:36.005 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2023-08-25T02:36:36.005 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.005 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.005 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.006 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.006 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.006 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.006 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.006 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2023-08-25T02:36:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2023-08-25T02:36:36.008 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2023-08-25T02:36:36.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.008 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.008 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2023-08-25T02:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2023-08-25T02:36:36.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.011 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.011 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.012 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.012 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.012 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.012 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.013 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.013 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.013 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.014 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2023-08-25T02:36:36.014 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2023-08-25T02:36:36.014 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.014 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.015 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.015 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-08-25T02:36:36.015 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.015 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.016 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.016 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.016 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.016 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.016 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.017 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:36.017 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:36.017 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.017 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.018 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.018 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.018 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.018 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.019 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.019 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.019 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.020 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:36.020 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:36.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.020 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.021 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2023-08-25T02:36:36.021 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2023-08-25T02:36:36.021 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.021 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.021 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.022 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2023-08-25T02:36:36.023 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2023-08-25T02:36:36.023 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.023 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.023 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.023 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2023-08-25T02:36:36.024 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.024 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.024 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.024 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.024 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2023-08-25T02:36:36.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.026 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.026 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.026 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.027 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.027 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.027 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.027 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.028 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.028 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2023-08-25T02:36:36.028 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2023-08-25T02:36:36.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.029 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.030 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.030 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.030 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.030 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.031 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.031 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.031 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.032 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.032 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.032 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.032 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.032 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.033 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.033 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.033 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.033 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.033 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.036 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.036 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.036 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.036 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.036 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.037 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.037 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.037 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.038 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.039 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.039 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.039 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.039 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.039 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.040 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.040 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.040 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.040 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.040 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.041 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.041 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.041 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.041 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.042 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.042 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.042 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.042 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.042 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.043 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2023-08-25T02:36:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2023-08-25T02:36:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2023-08-25T02:36:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2023-08-25T02:36:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2023-08-25T02:36:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2023-08-25T02:36:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-08-25T02:36:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2023-08-25T02:36:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2023-08-25T02:36:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2023-08-25T02:36:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2023-08-25T02:36:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2023-08-25T02:36:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2023-08-25T02:36:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-08-25T02:36:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2023-08-25T02:36:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2023-08-25T02:36:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2023-08-25T02:36:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2023-08-25T02:36:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2023-08-25T02:36:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2023-08-25T02:36:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2023-08-25T02:36:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2023-08-25T02:36:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2023-08-25T02:36:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2023-08-25T02:36:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2023-08-25T02:36:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-08-25T02:36:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "gid": 24896, 2023-08-25T02:36:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "z", 2023-08-25T02:36:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_features": 4540138322906710015, 2023-08-25T02:36:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2023-08-25T02:36:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2023-08-25T02:36:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2023-08-25T02:36:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2023-08-25T02:36:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2023-08-25T02:36:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2023-08-25T02:36:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2023-08-25T02:36:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2023-08-25T02:36:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2023-08-25T02:36:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2023-08-25T02:36:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2023-08-25T02:36:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2023-08-25T02:36:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2023-08-25T02:36:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2023-08-25T02:36:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2023-08-25T02:36:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2023-08-25T02:36:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2023-08-25T02:36:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2023-08-25T02:36:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2023-08-25T02:36:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2023-08-25T02:36:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2023-08-25T02:36:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2023-08-25T02:36:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2023-08-25T02:36:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2023-08-25T02:36:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2023-08-25T02:36:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2023-08-25T02:36:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2023-08-25T02:36:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2023-08-25T02:36:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2023-08-25T02:36:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2023-08-25T02:36:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2023-08-25T02:36:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2023-08-25T02:36:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2023-08-25T02:36:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2023-08-25T02:36:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2023-08-25T02:36:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2023-08-25T02:36:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2023-08-25T02:36:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2023-08-25T02:36:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-08-25T02:36:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2023-08-25T02:36:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2023-08-25T02:36:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2023-08-25T02:36:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-08-25T02:36:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-08-25T02:36:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2023-08-25T02:36:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2023-08-25T02:36:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2023-08-25T02:36:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2023-08-25T02:36:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2023-08-25T02:36:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2023-08-25T02:36:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-08-25T02:36:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2023-08-25T02:36:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2023-08-25T02:36:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2023-08-25T02:36:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2023-08-25T02:36:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2023-08-25T02:36:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-08-25T02:36:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2023-08-25T02:36:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2023-08-25T02:36:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-08-25T02:36:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2023-08-25T02:36:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2023-08-25T02:36:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2023-08-25T02:36:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap" 2023-08-25T02:36:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2023-08-25T02:36:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2023-08-25T02:36:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2023-08-25T02:36:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-08-25T02:36:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2023-08-25T02:36:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2023-08-25T02:36:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2023-08-25T02:36:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-08-25T02:36:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2023-08-25T02:36:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2023-08-25T02:36:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-08-25T02:36:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2023-08-25T02:36:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2023-08-25T02:36:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2023-08-25T02:36:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-08-25T02:36:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2023-08-25T02:36:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2023-08-25T02:36:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2023-08-25T02:36:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-08-25T02:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2023-08-25T02:36:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2023-08-25T02:36:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-08-25T02:36:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2023-08-25T02:36:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2023-08-25T02:36:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-08-25T02:36:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-08-25T02:36:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2023-08-25T02:36:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2023-08-25T02:36:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2023-08-25T02:36:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2023-08-25T02:36:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2023-08-25T02:36:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2023-08-25T02:36:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-08-25T02:36:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2023-08-25T02:36:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2023-08-25T02:36:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-08-25T02:36:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2023-08-25T02:36:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-08-25T02:36:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2023-08-25T02:36:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2023-08-25T02:36:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-08-25T02:36:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2023-08-25T02:36:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2023-08-25T02:36:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2023-08-25T02:36:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2023-08-25T02:36:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2023-08-25T02:36:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2023-08-25T02:36:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-08-25T02:36:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2023-08-25T02:36:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2023-08-25T02:36:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-08-25T02:36:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2023-08-25T02:36:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2023-08-25T02:36:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-08-25T02:36:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2023-08-25T02:36:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2023-08-25T02:36:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-08-25T02:36:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2023-08-25T02:36:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2023-08-25T02:36:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2023-08-25T02:36:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-08-25T02:36:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2023-08-25T02:36:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2023-08-25T02:36:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2023-08-25T02:36:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-08-25T02:36:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2023-08-25T02:36:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2023-08-25T02:36:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2023-08-25T02:36:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-08-25T02:36:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2023-08-25T02:36:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2023-08-25T02:36:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2023-08-25T02:36:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-08-25T02:36:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2023-08-25T02:36:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2023-08-25T02:36:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2023-08-25T02:36:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-08-25T02:36:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2023-08-25T02:36:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2023-08-25T02:36:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2023-08-25T02:36:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2023-08-25T02:36:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2023-08-25T02:36:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2023-08-25T02:36:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-08-25T02:36:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2023-08-25T02:36:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2023-08-25T02:36:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-08-25T02:36:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2023-08-25T02:36:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2023-08-25T02:36:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2023-08-25T02:36:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-08-25T02:36:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2023-08-25T02:36:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2023-08-25T02:36:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2023-08-25T02:36:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2023-08-25T02:36:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2023-08-25T02:36:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2023-08-25T02:36:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-08-25T02:36:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2023-08-25T02:36:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-08-25T02:36:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2023-08-25T02:36:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2023-08-25T02:36:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2023-08-25T02:36:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-08-25T02:36:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2023-08-25T02:36:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2023-08-25T02:36:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2023-08-25T02:36:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2023-08-25T02:36:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-25T02:36:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2023-08-25T02:36:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2023-08-25T02:36:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2023-08-25T02:36:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2023-08-25T02:36:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.312 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.312 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.312 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.312 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.313 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.313 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.313 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2023-08-25T02:36:36.314 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2023-08-25T02:36:36.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.314 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.314 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.314 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.315 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.315 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.315 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.315 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-08-25T02:36:36.315 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.316 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.316 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.316 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.316 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.317 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.317 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.317 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.317 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.318 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.318 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.318 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-08-25T02:36:36.318 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.318 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.319 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.319 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.319 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.319 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.320 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.320 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.320 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.320 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.320 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.321 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.321 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.321 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.321 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.321 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.322 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.322 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.322 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.322 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.324 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.324 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.324 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.324 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.324 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.325 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.325 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.325 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.325 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.325 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.326 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.326 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.326 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2023-08-25T02:36:36.327 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-08-25T02:36:36.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.327 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.327 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.328 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.328 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.328 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.328 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-08-25T02:36:36.328 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.329 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.329 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-08-25T02:36:36.329 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-08-25T02:36:36.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.330 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.330 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2023-08-25T02:36:36.331 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.331 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.331 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.331 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-08-25T02:36:36.331 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.332 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.332 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2023-08-25T02:36:36.332 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2023-08-25T02:36:36.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.333 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.333 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.333 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.334 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.334 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.334 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2023-08-25T02:36:36.334 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.334 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.335 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.335 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2023-08-25T02:36:36.335 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2023-08-25T02:36:36.335 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.336 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.336 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.336 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.337 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.337 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.337 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-08-25T02:36:36.337 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.337 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.338 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.339 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.339 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.339 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.339 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.340 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.340 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2023-08-25T02:36:36.340 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.341 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.341 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:36.341 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:36.341 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2023-08-25T02:36:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.343 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.343 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2023-08-25T02:36:36.343 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2023-08-25T02:36:36.343 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.343 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2023-08-25T02:36:36.344 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.344 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.344 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.344 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2023-08-25T02:36:36.345 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2023-08-25T02:36:36.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.345 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.345 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.345 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-08-25T02:36:36.346 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.346 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.346 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.346 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-08-25T02:36:36.346 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.347 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.347 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.347 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2023-08-25T02:36:36.347 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2023-08-25T02:36:36.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.348 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.348 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.348 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.349 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.349 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.349 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-08-25T02:36:36.349 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.350 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.350 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2023-08-25T02:36:36.350 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2023-08-25T02:36:36.350 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.351 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.351 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.351 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.351 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.351 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.352 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.352 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-08-25T02:36:36.352 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.352 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.353 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2023-08-25T02:36:36.353 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2023-08-25T02:36:36.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.353 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.354 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.354 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.354 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.354 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.354 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.355 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-08-25T02:36:36.355 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.355 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.356 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.357 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2023-08-25T02:36:36.357 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2023-08-25T02:36:36.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.358 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.358 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.359 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.359 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.359 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.360 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.360 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-08-25T02:36:36.361 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.361 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.361 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.362 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2023-08-25T02:36:36.362 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2023-08-25T02:36:36.363 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.363 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.364 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.364 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.365 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.365 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.365 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-08-25T02:36:36.366 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.366 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.367 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.367 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2023-08-25T02:36:36.368 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2023-08-25T02:36:36.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.368 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.369 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.369 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2023-08-25T02:36:36.370 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.370 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.370 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.371 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2023-08-25T02:36:36.371 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.372 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.373 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2023-08-25T02:36:36.373 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2023-08-25T02:36:36.374 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.374 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.374 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.375 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.375 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.375 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.376 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.376 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-08-25T02:36:36.377 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.377 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.377 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.378 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2023-08-25T02:36:36.378 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2023-08-25T02:36:36.379 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.379 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.381 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.381 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.381 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-08-25T02:36:36.382 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2023-08-25T02:36:36.384 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2023-08-25T02:36:36.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.384 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.385 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.385 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.385 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.385 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-08-25T02:36:36.385 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.386 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.386 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.386 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2023-08-25T02:36:36.386 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2023-08-25T02:36:36.387 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.387 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.387 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.387 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.387 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.388 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.388 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.388 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2023-08-25T02:36:36.388 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.388 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.389 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.389 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2023-08-25T02:36:36.389 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2023-08-25T02:36:36.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.390 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.390 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.390 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.390 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2023-08-25T02:36:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2023-08-25T02:36:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.393 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.393 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.393 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.393 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.393 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-08-25T02:36:36.394 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.394 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.394 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.395 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.396 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.396 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.396 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.396 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.397 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.397 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.397 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.397 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.397 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.398 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.399 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.399 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.399 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.399 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.399 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.400 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.400 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.401 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.402 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.402 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.402 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.402 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.403 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.403 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.403 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.404 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.404 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.404 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.404 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.405 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.405 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.405 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.405 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.405 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.406 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.407 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.407 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.407 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.408 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.408 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.408 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.408 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.409 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.409 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.409 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.409 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2023-08-25T02:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2023-08-25T02:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.411 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2023-08-25T02:36:36.411 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.411 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.411 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.411 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2023-08-25T02:36:36.412 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2023-08-25T02:36:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2023-08-25T02:36:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.414 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.414 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.414 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2023-08-25T02:36:36.414 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.414 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.415 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2023-08-25T02:36:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.416 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-08-25T02:36:36.417 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-08-25T02:36:36.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.417 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.417 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.418 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.418 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.418 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.418 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.418 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2023-08-25T02:36:36.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.420 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.420 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.420 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.420 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.421 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.421 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.421 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.421 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.421 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-08-25T02:36:36.422 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-08-25T02:36:36.422 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.422 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.423 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2023-08-25T02:36:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2023-08-25T02:36:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.427 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.427 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.428 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.428 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.428 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.429 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.429 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.430 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.431 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-08-25T02:36:36.431 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-08-25T02:36:36.431 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.431 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.432 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.432 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.432 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.432 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.432 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.433 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.433 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.433 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.433 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.434 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2023-08-25T02:36:36.434 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2023-08-25T02:36:36.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.434 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.435 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.435 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.435 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.435 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.435 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.436 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.436 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.436 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.436 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-08-25T02:36:36.437 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-08-25T02:36:36.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.437 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.437 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.437 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.438 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.438 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.438 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.438 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.438 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.439 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.439 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.439 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-08-25T02:36:36.439 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-08-25T02:36:36.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.440 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.440 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.440 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.441 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.441 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.441 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.441 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.441 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.442 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.442 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-08-25T02:36:36.442 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-08-25T02:36:36.442 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.443 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.443 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.443 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.443 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.444 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.444 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.444 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.444 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.444 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.445 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-08-25T02:36:36.445 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-08-25T02:36:36.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.446 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.446 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.446 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.446 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.446 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.447 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2023-08-25T02:36:36.448 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-08-25T02:36:36.448 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.448 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.448 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.448 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.449 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.449 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.449 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.449 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.449 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.450 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.450 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.450 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2023-08-25T02:36:36.450 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-08-25T02:36:36.451 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.451 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.451 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.451 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.453 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.453 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-25T02:36:36.453 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-08-25T02:36:36.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.454 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.454 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.454 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.454 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.456 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-08-25T02:36:36.456 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-08-25T02:36:36.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.456 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.456 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.457 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.457 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.457 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.457 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.457 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.458 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.458 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.458 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2023-08-25T02:36:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2023-08-25T02:36:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.459 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:36.460 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.460 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.460 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.460 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.460 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.461 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.461 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-08-25T02:36:36.461 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-08-25T02:36:36.462 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.462 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.462 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.462 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.462 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.463 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.464 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2023-08-25T02:36:36.464 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2023-08-25T02:36:36.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.464 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.465 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.465 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.465 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.465 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.465 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.466 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.466 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.466 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.466 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.467 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2023-08-25T02:36:36.467 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2023-08-25T02:36:36.467 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.467 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.467 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.468 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:36.468 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.468 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.468 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.468 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.469 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.469 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.469 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.469 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.470 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.471 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.471 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.471 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.471 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.471 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.472 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.472 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-08-25T02:36:36.472 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-08-25T02:36:36.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.473 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.473 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.473 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.473 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.473 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.474 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.474 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.474 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.474 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.474 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.476 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.476 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.476 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.476 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.477 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.477 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.477 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.477 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.478 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2023-08-25T02:36:36.478 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-08-25T02:36:36.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.478 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.479 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.480 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.480 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.480 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2023-08-25T02:36:36.480 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2023-08-25T02:36:36.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.481 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.481 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.481 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.482 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.482 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.482 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.482 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.482 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.483 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.483 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-08-25T02:36:36.483 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-08-25T02:36:36.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.484 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.484 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.484 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.484 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.484 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.485 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.485 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.485 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.485 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.487 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.487 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.487 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.487 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.487 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.488 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.488 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.488 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.488 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-08-25T02:36:36.489 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-08-25T02:36:36.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.489 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.489 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.489 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.490 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.490 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.490 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.490 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.490 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.491 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.491 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-08-25T02:36:36.491 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-08-25T02:36:36.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.492 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.492 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.492 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.493 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-08-25T02:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-08-25T02:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-08-25T02:36:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-08-25T02:36:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-08-25T02:36:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-08-25T02:36:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2023-08-25T02:36:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2023-08-25T02:36:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-08-25T02:36:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-08-25T02:36:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-08-25T02:36:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-08-25T02:36:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-08-25T02:36:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-08-25T02:36:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-08-25T02:36:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2023-08-25T02:36:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2023-08-25T02:36:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2023-08-25T02:36:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2023-08-25T02:36:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-08-25T02:36:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-08-25T02:36:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2023-08-25T02:36:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2023-08-25T02:36:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-08-25T02:36:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-08-25T02:36:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-08-25T02:36:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-08-25T02:36:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-08-25T02:36:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-08-25T02:36:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2023-08-25T02:36:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2023-08-25T02:36:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2023-08-25T02:36:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2023-08-25T02:36:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2023-08-25T02:36:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2023-08-25T02:36:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2023-08-25T02:36:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2023-08-25T02:36:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2023-08-25T02:36:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2023-08-25T02:36:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2023-08-25T02:36:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2023-08-25T02:36:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2023-08-25T02:36:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2023-08-25T02:36:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2023-08-25T02:36:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2023-08-25T02:36:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:36.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.587 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.588 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.588 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.588 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.589 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.589 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.590 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.590 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.590 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.591 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2023-08-25T02:36:36.591 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2023-08-25T02:36:36.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.591 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.592 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2023-08-25T02:36:36.592 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.592 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.594 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:36.594 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:36.594 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.594 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.595 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2023-08-25T02:36:36.595 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.595 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.595 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.596 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.596 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2023-08-25T02:36:36.596 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.596 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.596 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.597 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.597 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:36.597 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:36.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.598 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.598 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:36.598 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:36.599 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:36.599 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.599 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.599 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.600 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.600 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2023-08-25T02:36:36.600 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2023-08-25T02:36:36.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.601 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.602 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.602 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.602 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.603 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2023-08-25T02:36:36.603 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.604 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.604 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.604 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2023-08-25T02:36:36.604 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2023-08-25T02:36:36.604 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.605 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.605 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.605 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.605 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.607 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.607 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.607 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.608 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.608 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.608 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.608 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.608 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.609 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.609 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.609 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.609 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.609 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.610 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.610 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.610 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.610 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.610 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.612 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.612 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.612 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.612 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.613 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.613 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.613 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.614 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.615 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.615 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.615 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.615 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.616 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.616 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.616 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.617 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.617 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.617 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.618 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.618 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.618 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.618 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.619 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.619 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.620 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.620 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.620 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.620 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.621 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.621 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.621 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.622 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.622 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2023-08-25T02:36:36.622 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2023-08-25T02:36:36.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.623 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.623 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2023-08-25T02:36:36.623 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.624 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.624 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.624 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-08-25T02:36:36.624 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.624 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.625 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.625 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2023-08-25T02:36:36.625 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2023-08-25T02:36:36.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.626 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.627 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2023-08-25T02:36:36.627 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.627 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.628 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2023-08-25T02:36:36.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2023-08-25T02:36:36.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.628 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.628 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.629 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2023-08-25T02:36:36.629 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.629 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.629 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.629 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2023-08-25T02:36:36.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.630 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.630 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.630 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2023-08-25T02:36:36.631 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2023-08-25T02:36:36.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.631 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.632 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:36.632 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.632 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.633 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.633 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2023-08-25T02:36:36.633 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.634 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.634 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2023-08-25T02:36:36.635 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2023-08-25T02:36:36.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.636 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.636 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.637 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.637 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.637 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.637 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-08-25T02:36:36.638 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.638 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.638 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:36.639 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:36.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.639 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.640 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:36.640 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.640 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.640 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.640 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2023-08-25T02:36:36.641 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.641 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.641 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2023-08-25T02:36:36.642 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2023-08-25T02:36:36.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.642 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.642 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.643 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2023-08-25T02:36:36.643 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.643 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.643 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.643 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-08-25T02:36:36.644 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.644 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.644 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.645 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.645 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2023-08-25T02:36:36.645 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.646 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.646 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.646 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.646 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.647 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.647 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.647 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.647 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.648 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.648 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.648 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.648 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.648 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.649 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.649 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.649 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.649 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.649 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.650 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.650 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.650 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.651 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.651 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.651 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.651 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.651 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.652 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.652 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.652 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.652 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.652 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.654 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.654 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.654 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.655 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.655 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.655 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.655 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.655 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.656 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.656 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.656 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.657 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.657 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.657 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.657 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.659 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.659 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.659 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.659 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.659 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.661 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2023-08-25T02:36:36.662 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2023-08-25T02:36:36.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.662 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.663 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:36.663 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.663 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.663 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.664 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.664 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.664 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.665 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2023-08-25T02:36:36.665 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2023-08-25T02:36:36.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.665 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.666 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2023-08-25T02:36:36.666 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.666 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.666 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.666 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.667 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.667 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.668 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:36.668 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:36.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.669 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.669 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.669 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:36.669 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.669 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.670 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.670 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.670 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.670 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.671 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.671 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.671 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.672 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2023-08-25T02:36:36.672 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2023-08-25T02:36:36.672 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2023-08-25T02:36:36.672 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.673 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2023-08-25T02:36:36.673 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2023-08-25T02:36:36.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.674 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.674 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2023-08-25T02:36:36.674 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.675 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.675 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.675 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-08-25T02:36:36.676 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.676 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.676 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.677 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2023-08-25T02:36:36.677 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2023-08-25T02:36:36.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.677 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.677 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.678 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2023-08-25T02:36:36.678 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.678 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.679 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.679 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-08-25T02:36:36.679 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.679 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.679 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.680 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2023-08-25T02:36:36.680 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2023-08-25T02:36:36.680 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.681 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.681 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.681 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.682 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.682 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.682 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2023-08-25T02:36:36.682 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.682 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.683 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.683 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:36.683 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:36.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.684 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.684 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2023-08-25T02:36:36.684 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2023-08-25T02:36:36.684 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.686 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.686 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.686 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.686 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.686 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.687 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.687 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.687 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.687 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.687 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.688 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.688 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.688 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.688 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.688 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.689 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.689 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.689 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.689 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.690 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.690 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.690 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.690 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.691 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.692 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.692 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.693 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.694 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.694 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.694 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.694 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.694 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.695 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.695 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.695 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.695 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.695 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.696 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.696 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.696 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.697 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.697 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.697 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.697 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.697 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.699 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.699 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.699 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2023-08-25T02:36:36.700 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2023-08-25T02:36:36.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.700 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.700 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.701 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.701 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.701 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.701 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2023-08-25T02:36:36.701 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2023-08-25T02:36:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2023-08-25T02:36:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2023-08-25T02:36:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2023-08-25T02:36:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2023-08-25T02:36:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2023-08-25T02:36:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-08-25T02:36:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2023-08-25T02:36:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2023-08-25T02:36:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2023-08-25T02:36:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2023-08-25T02:36:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2023-08-25T02:36:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.715 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2023-08-25T02:36:36.715 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.715 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.715 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.715 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:36.716 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.716 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.716 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2023-08-25T02:36:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2023-08-25T02:36:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2023-08-25T02:36:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2023-08-25T02:36:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2023-08-25T02:36:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2023-08-25T02:36:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2023-08-25T02:36:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2023-08-25T02:36:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2023-08-25T02:36:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2023-08-25T02:36:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-08-25T02:36:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2023-08-25T02:36:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2023-08-25T02:36:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2023-08-25T02:36:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2023-08-25T02:36:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2023-08-25T02:36:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2023-08-25T02:36:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2023-08-25T02:36:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2023-08-25T02:36:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2023-08-25T02:36:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2023-08-25T02:36:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2023-08-25T02:36:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2023-08-25T02:36:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2023-08-25T02:36:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2023-08-25T02:36:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2023-08-25T02:36:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2023-08-25T02:36:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2023-08-25T02:36:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2023-08-25T02:36:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2023-08-25T02:36:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-08-25T02:36:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2023-08-25T02:36:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.825 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.825 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.825 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.825 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.827 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.827 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.827 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.827 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.827 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.828 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.828 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.828 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.829 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.829 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.829 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.830 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.830 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.830 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.830 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.830 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.831 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.831 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.831 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.831 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.831 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.832 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.832 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.832 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.835 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.835 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.836 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2023-08-25T02:36:36.836 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.836 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.836 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.837 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.838 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.838 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.839 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.839 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.839 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.839 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.839 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.840 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.840 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.840 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.840 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.841 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.841 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.841 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.842 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.842 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.842 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.842 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.844 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.844 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.844 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.845 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.845 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.845 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.846 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.846 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.846 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.846 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.846 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.847 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.847 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.847 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.847 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.847 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.848 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.848 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.848 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.849 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.849 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.849 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.850 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.850 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.850 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.850 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.851 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.852 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:36.852 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.852 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.852 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.852 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2023-08-25T02:36:36.853 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2023-08-25T02:36:36.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.853 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.853 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.854 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.854 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.854 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.854 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-08-25T02:36:36.854 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.855 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.855 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.855 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.855 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.856 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.856 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.856 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.856 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.857 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.857 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.857 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.857 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.857 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.858 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.858 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.858 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.858 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.858 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.859 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.860 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.860 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.861 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.861 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.861 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.861 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.861 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.862 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.862 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.862 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.863 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.863 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.863 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.863 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.863 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.864 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.864 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.864 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.864 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.864 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.865 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.865 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.865 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.865 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.865 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.866 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.866 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.866 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.866 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.867 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.868 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.868 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.868 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.868 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.868 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.869 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.869 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.869 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2023-08-25T02:36:36.869 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:36.870 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.870 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.870 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.870 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.870 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.871 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.871 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.871 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2023-08-25T02:36:36.871 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2023-08-25T02:36:36.871 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2023-08-25T02:36:36.872 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.872 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2023-08-25T02:36:36.872 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.872 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.872 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.873 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2023-08-25T02:36:36.874 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.874 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.874 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.874 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.874 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.876 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.876 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.876 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.877 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.877 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.877 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.877 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.878 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.878 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.878 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.878 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.878 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.879 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.879 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.879 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.880 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.880 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.880 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.881 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.881 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.881 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.881 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.881 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.882 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.882 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.882 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.882 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.883 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.883 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.883 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.883 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.883 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.884 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.885 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.885 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.885 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.885 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.885 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.886 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.886 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.886 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.887 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.887 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.887 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.887 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.887 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.888 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.888 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.888 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.888 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.888 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.889 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.889 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2023-08-25T02:36:36.890 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.891 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.891 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.891 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.891 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.892 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.892 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.893 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.893 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.893 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.893 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.893 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.894 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.894 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.894 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.894 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.894 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.895 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.895 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.895 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.895 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.896 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.896 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.896 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.898 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.899 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.899 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.899 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.899 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.899 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.900 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.900 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.900 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.900 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.901 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.901 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.901 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.901 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.901 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.902 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.902 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.902 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.903 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.903 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.903 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.903 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.903 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.904 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.904 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.904 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.904 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.904 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.905 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.905 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.906 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2023-08-25T02:36:36.906 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.906 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.908 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.908 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.908 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.908 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.909 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.909 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.909 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.909 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.909 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.910 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.910 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.910 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.910 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.910 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.911 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.911 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.912 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.912 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.912 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.912 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.913 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.913 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.913 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.913 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.914 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.915 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.915 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.915 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.915 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.915 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.916 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.916 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.916 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.916 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.916 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.917 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.917 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.917 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.917 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.917 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.918 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.918 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.919 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.920 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.920 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.920 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.921 INFO:teuthology.orchestra.run.smithi089.stdout: "noautoscale": { 2023-08-25T02:36:36.921 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "noautoscale", 2023-08-25T02:36:36.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.921 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.921 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.922 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.922 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.922 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.922 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.922 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "global autoscale flag", 2023-08-25T02:36:36.923 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-08-25T02:36:36.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.923 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.923 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:36.924 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:36.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.924 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.924 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:36.925 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.925 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.925 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.925 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.925 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.926 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.926 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.926 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2023-08-25T02:36:36.927 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2023-08-25T02:36:36.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.927 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.927 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:36.928 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2023-08-25T02:36:36.928 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.928 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.928 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2023-08-25T02:36:36.929 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-08-25T02:36:36.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.929 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.930 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2023-08-25T02:36:36.930 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.931 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.931 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.931 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2023-08-25T02:36:36.931 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2023-08-25T02:36:36.931 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.932 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.932 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.932 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.932 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.933 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.933 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2023-08-25T02:36:36.933 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.933 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.933 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.934 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:36.934 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:36.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.934 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.936 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.937 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.937 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.938 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.938 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.938 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.938 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.938 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.939 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.939 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.939 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.939 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.939 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.940 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.940 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.940 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.940 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.941 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.941 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.941 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.942 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.942 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.942 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.942 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.942 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.943 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.943 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.943 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.943 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2023-08-25T02:36:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2023-08-25T02:36:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2023-08-25T02:36:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2023-08-25T02:36:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2023-08-25T02:36:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2023-08-25T02:36:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2023-08-25T02:36:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2023-08-25T02:36:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-08-25T02:36:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-08-25T02:36:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2023-08-25T02:36:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2023-08-25T02:36:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2023-08-25T02:36:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-08-25T02:36:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2023-08-25T02:36:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2023-08-25T02:36:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2023-08-25T02:36:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2023-08-25T02:36:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.985 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.985 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.985 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.985 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.985 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.986 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:36.986 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:36.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-08-25T02:36:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2023-08-25T02:36:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-08-25T02:36:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2023-08-25T02:36:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2023-08-25T02:36:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2023-08-25T02:36:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2023-08-25T02:36:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2023-08-25T02:36:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2023-08-25T02:36:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2023-08-25T02:36:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2023-08-25T02:36:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2023-08-25T02:36:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2023-08-25T02:36:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2023-08-25T02:36:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.021 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.021 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2023-08-25T02:36:37.021 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2023-08-25T02:36:37.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.022 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.023 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.023 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.023 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.023 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.024 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2023-08-25T02:36:37.024 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.025 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.025 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.025 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2023-08-25T02:36:37.025 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2023-08-25T02:36:37.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.026 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.026 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.026 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.026 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.027 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.027 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.027 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.027 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.028 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:37.028 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:37.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.028 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.029 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.029 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.029 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.029 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.029 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.030 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.030 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.030 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.030 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.031 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.031 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.031 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.032 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.032 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.032 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.032 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.032 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.033 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.033 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.033 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.033 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.033 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.034 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.034 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.034 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.034 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.034 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.035 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.035 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.035 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.036 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.036 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.036 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.036 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.037 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.037 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.037 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.038 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.039 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.039 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.039 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.039 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.039 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.040 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.040 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.040 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.040 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.041 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.041 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.041 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.041 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.041 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.042 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.042 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.042 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.042 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.043 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.043 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.043 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.043 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.044 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.044 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.044 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.045 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:37.045 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:37.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.045 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.046 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.046 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.046 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.046 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.046 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.047 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.047 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.047 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.047 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:37.048 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:37.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.048 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.048 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.049 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.049 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.049 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.049 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.050 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.050 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.051 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2023-08-25T02:36:37.051 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.052 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.052 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.052 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.052 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.052 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.053 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.053 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.053 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.054 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.054 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.054 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.054 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.054 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.055 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.055 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.055 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.055 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.055 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.056 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.056 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.057 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.057 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.057 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.058 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.058 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.058 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.058 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.059 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.059 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.059 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.059 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.060 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.060 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.060 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.061 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.062 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.062 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.062 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.062 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.062 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.065 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.065 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.065 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.065 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.066 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.066 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.066 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.066 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.066 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2023-08-25T02:36:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.068 INFO:teuthology.orchestra.run.smithi089.stdout: "drive_group_interval": { 2023-08-25T02:36:37.069 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "drive_group_interval", 2023-08-25T02:36:37.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.069 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.070 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300.0", 2023-08-25T02:36:37.070 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.070 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.070 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.070 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-08-25T02:36:37.071 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.071 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.071 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.072 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.072 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.072 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.072 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.073 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.073 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.073 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.073 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.073 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.074 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.074 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.074 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.074 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.074 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.075 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.075 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.075 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.075 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.076 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.076 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.076 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.076 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.077 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.077 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.077 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.077 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.077 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.078 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.079 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.079 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.079 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.080 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.080 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.080 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.080 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.080 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.081 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.081 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.081 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.081 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.081 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.082 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.082 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.082 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.083 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.084 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.084 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.084 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.084 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.085 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.085 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2023-08-25T02:36:37.085 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2023-08-25T02:36:37.085 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.086 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.086 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2023-08-25T02:36:37.086 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.086 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.087 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.087 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-08-25T02:36:37.087 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.087 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.087 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.088 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2023-08-25T02:36:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.090 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.090 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.090 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.091 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.091 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.091 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.091 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.091 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.092 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.092 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.092 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.092 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.092 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.093 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.094 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.094 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.095 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.095 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.095 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.095 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.095 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.096 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.096 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.096 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.096 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.098 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.098 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.098 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.098 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.099 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.099 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.099 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.099 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.099 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.100 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.100 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.100 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.100 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.101 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.101 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.101 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.102 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.102 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.102 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.103 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.103 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.103 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2023-08-25T02:36:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2023-08-25T02:36:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2023-08-25T02:36:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2023-08-25T02:36:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2023-08-25T02:36:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2023-08-25T02:36:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2023-08-25T02:36:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2023-08-25T02:36:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2023-08-25T02:36:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2023-08-25T02:36:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2023-08-25T02:36:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2023-08-25T02:36:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2023-08-25T02:36:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2023-08-25T02:36:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2023-08-25T02:36:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2023-08-25T02:36:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2023-08-25T02:36:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2023-08-25T02:36:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2023-08-25T02:36:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2023-08-25T02:36:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2023-08-25T02:36:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2023-08-25T02:36:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2023-08-25T02:36:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2023-08-25T02:36:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2023-08-25T02:36:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2023-08-25T02:36:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2023-08-25T02:36:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2023-08-25T02:36:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2023-08-25T02:36:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2023-08-25T02:36:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2023-08-25T02:36:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.171 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.171 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.171 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.171 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.171 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.172 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.172 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.172 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.172 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.172 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2023-08-25T02:36:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2023-08-25T02:36:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2023-08-25T02:36:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2023-08-25T02:36:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-08-25T02:36:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2023-08-25T02:36:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.204 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.204 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.204 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.204 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.204 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.205 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.205 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.205 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.205 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.206 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.206 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.206 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.207 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.207 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.207 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.207 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.207 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.208 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.208 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.208 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.208 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.208 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.209 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.209 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.209 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.210 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.210 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.210 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.210 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.211 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.211 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.211 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.211 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.211 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.212 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.213 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2023-08-25T02:36:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2023-08-25T02:36:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.215 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.215 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.215 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.215 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2023-08-25T02:36:37.215 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.216 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.216 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2023-08-25T02:36:37.216 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2023-08-25T02:36:37.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.217 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.217 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.217 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.218 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.218 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.218 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-08-25T02:36:37.218 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.219 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.220 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.220 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.220 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.220 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.221 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-08-25T02:36:37.221 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.221 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.222 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2023-08-25T02:36:37.222 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2023-08-25T02:36:37.222 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.222 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.223 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.223 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.223 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.223 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.223 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-08-25T02:36:37.224 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.224 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.224 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.224 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2023-08-25T02:36:37.225 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2023-08-25T02:36:37.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.225 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.225 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.227 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.227 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2023-08-25T02:36:37.227 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2023-08-25T02:36:37.227 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.228 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.228 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.228 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.228 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.228 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.230 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2023-08-25T02:36:37.230 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2023-08-25T02:36:37.230 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.230 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.231 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.231 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.231 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.231 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.231 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.232 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.232 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.232 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.233 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2023-08-25T02:36:37.233 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2023-08-25T02:36:37.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.233 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.234 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.235 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.235 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.235 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:37.235 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.238 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.238 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:37.238 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:37.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.239 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.239 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.239 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2023-08-25T02:36:37.239 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2023-08-25T02:36:37.239 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.240 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.240 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.240 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.240 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.240 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2023-08-25T02:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2023-08-25T02:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.242 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.243 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.243 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.243 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.243 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2023-08-25T02:36:37.244 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2023-08-25T02:36:37.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.244 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.244 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.244 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.245 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.245 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.245 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.245 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.245 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.246 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.246 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.246 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2023-08-25T02:36:37.246 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2023-08-25T02:36:37.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.247 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.247 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.247 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.247 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.249 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.249 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.249 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.250 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.250 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.250 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.250 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.251 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.251 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.251 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.251 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.251 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.252 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.252 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.252 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.252 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.252 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.253 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2023-08-25T02:36:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2023-08-25T02:36:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2023-08-25T02:36:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2023-08-25T02:36:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2023-08-25T02:36:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2023-08-25T02:36:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-08-25T02:36:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2023-08-25T02:36:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2023-08-25T02:36:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2023-08-25T02:36:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2023-08-25T02:36:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2023-08-25T02:36:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2023-08-25T02:36:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2023-08-25T02:36:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2023-08-25T02:36:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-08-25T02:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2023-08-25T02:36:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2023-08-25T02:36:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2023-08-25T02:36:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2023-08-25T02:36:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2023-08-25T02:36:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.327 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.327 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.327 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.327 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.328 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.328 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.328 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.328 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2023-08-25T02:36:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2023-08-25T02:36:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2023-08-25T02:36:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2023-08-25T02:36:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2023-08-25T02:36:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2023-08-25T02:36:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2023-08-25T02:36:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-08-25T02:36:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "modules": [ 2023-08-25T02:36:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2023-08-25T02:36:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "dashboard", 2023-08-25T02:36:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "iostat", 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "nfs", 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "restful", 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_schedule" 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2023-08-25T02:36:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2023-08-25T02:36:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2023-08-25T02:36:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.347 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.347 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.347 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.347 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.347 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.348 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.348 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.348 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.348 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.348 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.349 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.349 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.349 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.350 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.350 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.350 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.350 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.351 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.351 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.351 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.351 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.351 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.352 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.352 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.353 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.353 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.353 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.353 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.354 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.355 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.355 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.355 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.355 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.355 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.356 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.356 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.356 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.357 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.357 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.357 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.358 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.358 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.358 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.358 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.358 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.359 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.359 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.359 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.359 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2023-08-25T02:36:37.360 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2023-08-25T02:36:37.360 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.360 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.360 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.360 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.361 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.361 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.361 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.361 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2023-08-25T02:36:37.361 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2023-08-25T02:36:37.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.363 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.363 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.363 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2023-08-25T02:36:37.363 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.363 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2023-08-25T02:36:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.365 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2023-08-25T02:36:37.365 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2023-08-25T02:36:37.365 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.365 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.366 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.366 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.366 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.366 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.367 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2023-08-25T02:36:37.368 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2023-08-25T02:36:37.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.368 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.368 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2023-08-25T02:36:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.370 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.370 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.370 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2023-08-25T02:36:37.370 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.371 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.372 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.372 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2023-08-25T02:36:37.372 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.372 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.373 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2023-08-25T02:36:37.373 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2023-08-25T02:36:37.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.373 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.374 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.374 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.374 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.374 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.374 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2023-08-25T02:36:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2023-08-25T02:36:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2023-08-25T02:36:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.377 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.377 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.377 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.377 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.377 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2023-08-25T02:36:37.378 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.378 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.378 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.378 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.379 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.380 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.380 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.380 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2023-08-25T02:36:37.381 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.381 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.381 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.382 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2023-08-25T02:36:37.382 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.382 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.383 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.383 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2023-08-25T02:36:37.383 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2023-08-25T02:36:37.383 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.384 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.384 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.384 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.385 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.385 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2023-08-25T02:36:37.385 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.385 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.385 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.386 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2023-08-25T02:36:37.386 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2023-08-25T02:36:37.386 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.387 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.387 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.387 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2023-08-25T02:36:37.387 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.387 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.388 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.388 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2023-08-25T02:36:37.388 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:37.388 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.389 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.389 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2023-08-25T02:36:37.389 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2023-08-25T02:36:37.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:37.390 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.390 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:37.390 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:37.390 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:37.391 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.391 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-08-25T02:36:37.391 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:37.391 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.392 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.392 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2023-08-25T02:36:37.393 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2023-08-25T02:36:37.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:37.393 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.393 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2023-08-25T02:36:37.394 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:37.394 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2023-08-25T02:36:37.394 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.394 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-08-25T02:36:37.394 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.395 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.395 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.395 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2023-08-25T02:36:37.395 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2023-08-25T02:36:37.396 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.396 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.396 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.396 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2023-08-25T02:36:37.396 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.397 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.397 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.397 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-08-25T02:36:37.397 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-08-25T02:36:37.397 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.398 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.398 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2023-08-25T02:36:37.398 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2023-08-25T02:36:37.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2023-08-25T02:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2023-08-25T02:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2023-08-25T02:36:37.400 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.400 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2023-08-25T02:36:37.400 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-08-25T02:36:37.400 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.400 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.401 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2023-08-25T02:36:37.401 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2023-08-25T02:36:37.401 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.401 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.402 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2023-08-25T02:36:37.403 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-08-25T02:36:37.403 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.403 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.404 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2023-08-25T02:36:37.404 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2023-08-25T02:36:37.404 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:37.404 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.405 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:37.405 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2023-08-25T02:36:37.405 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2023-08-25T02:36:37.405 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.405 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-08-25T02:36:37.406 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-08-25T02:36:37.406 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.406 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.407 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.407 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.407 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.407 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.408 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.408 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.408 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.408 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.409 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.409 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.409 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.409 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.409 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.410 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.410 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.410 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.410 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.410 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.411 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.411 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.411 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.411 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.412 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.412 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.412 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.412 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.413 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.413 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.413 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.413 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.413 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.414 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.414 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.414 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.415 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.415 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.415 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.415 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.415 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.416 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.416 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.416 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.416 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.416 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.417 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.417 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.417 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.417 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.417 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.418 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.418 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.418 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.418 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.419 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.419 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.419 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.419 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.419 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.420 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.420 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.420 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.420 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.420 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.421 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.422 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:37.422 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.422 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.422 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.422 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-08-25T02:36:37.423 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.423 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.423 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.423 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:37.424 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:37.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.424 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.424 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2023-08-25T02:36:37.425 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.425 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.425 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.425 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2023-08-25T02:36:37.425 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2023-08-25T02:36:37.426 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap" 2023-08-25T02:36:37.426 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.426 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2023-08-25T02:36:37.426 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.426 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.427 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.428 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.428 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.428 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.428 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.429 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-08-25T02:36:37.429 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.429 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.429 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.430 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:37.430 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:37.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.430 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.431 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.431 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:37.431 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.431 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.431 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.432 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-08-25T02:36:37.432 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.432 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.432 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.433 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2023-08-25T02:36:37.433 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2023-08-25T02:36:37.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.433 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.434 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:37.434 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:37.434 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.434 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.434 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2023-08-25T02:36:37.435 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-08-25T02:36:37.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.435 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.435 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2023-08-25T02:36:37.436 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2023-08-25T02:36:37.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:37.436 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.436 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.437 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.437 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.437 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.437 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-08-25T02:36:37.437 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.438 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.438 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.438 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.438 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.439 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2023-08-25T02:36:37.439 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.439 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.439 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.440 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2023-08-25T02:36:37.440 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2023-08-25T02:36:37.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.440 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.441 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:37.441 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.441 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.441 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.441 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-08-25T02:36:37.442 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.442 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.442 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.442 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2023-08-25T02:36:37.443 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.444 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.444 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.444 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-08-25T02:36:37.444 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.444 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.445 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.445 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2023-08-25T02:36:37.445 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2023-08-25T02:36:37.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.446 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.446 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.446 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2023-08-25T02:36:37.446 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.446 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.447 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.447 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-08-25T02:36:37.447 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.447 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.447 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.448 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2023-08-25T02:36:37.448 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2023-08-25T02:36:37.448 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.448 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.450 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-08-25T02:36:37.450 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-08-25T02:36:37.450 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.450 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.451 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2023-08-25T02:36:37.451 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2023-08-25T02:36:37.451 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.451 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.452 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2023-08-25T02:36:37.453 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.453 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.453 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.454 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2023-08-25T02:36:37.454 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2023-08-25T02:36:37.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:37.454 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.455 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2023-08-25T02:36:37.455 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.455 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.455 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.455 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-08-25T02:36:37.456 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.456 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.456 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.456 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2023-08-25T02:36:37.457 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2023-08-25T02:36:37.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.457 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.458 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.458 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.458 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.458 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.458 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-08-25T02:36:37.459 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.459 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.459 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.459 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2023-08-25T02:36:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2023-08-25T02:36:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.461 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.461 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.461 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.461 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-08-25T02:36:37.461 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2023-08-25T02:36:37.462 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2023-08-25T02:36:37.463 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.463 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.463 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.463 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.464 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.464 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.464 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.464 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-08-25T02:36:37.464 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.465 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.466 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2023-08-25T02:36:37.466 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2023-08-25T02:36:37.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.466 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.467 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.467 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-08-25T02:36:37.467 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.467 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.468 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.468 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:37.468 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.468 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.469 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.469 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2023-08-25T02:36:37.469 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2023-08-25T02:36:37.469 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.470 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.470 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.470 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2023-08-25T02:36:37.470 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.471 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.472 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2023-08-25T02:36:37.472 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2023-08-25T02:36:37.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.473 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.473 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.473 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-08-25T02:36:37.473 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.473 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.474 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.474 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2023-08-25T02:36:37.474 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.474 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.474 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2023-08-25T02:36:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2023-08-25T02:36:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-08-25T02:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.478 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2023-08-25T02:36:37.478 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2023-08-25T02:36:37.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.478 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.479 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-08-25T02:36:37.479 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.479 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.479 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.480 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2023-08-25T02:36:37.480 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.480 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.480 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2023-08-25T02:36:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2023-08-25T02:36:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.481 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-08-25T02:36:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.482 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.483 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2023-08-25T02:36:37.483 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.483 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.484 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2023-08-25T02:36:37.484 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2023-08-25T02:36:37.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.484 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.485 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-08-25T02:36:37.485 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.485 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.485 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.486 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2023-08-25T02:36:37.486 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.486 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.486 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.487 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2023-08-25T02:36:37.487 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2023-08-25T02:36:37.487 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.487 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.487 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.488 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-08-25T02:36:37.488 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.488 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.488 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.489 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2023-08-25T02:36:37.489 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.489 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.489 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.490 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2023-08-25T02:36:37.490 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2023-08-25T02:36:37.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.490 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.491 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-08-25T02:36:37.491 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.491 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.491 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.491 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2023-08-25T02:36:37.492 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.492 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.492 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.493 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-08-25T02:36:37.494 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.494 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.494 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.494 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2023-08-25T02:36:37.494 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.495 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.495 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2023-08-25T02:36:37.495 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2023-08-25T02:36:37.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.496 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.496 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.496 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-08-25T02:36:37.496 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2023-08-25T02:36:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2023-08-25T02:36:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2023-08-25T02:36:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2023-08-25T02:36:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-08-25T02:36:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2023-08-25T02:36:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2023-08-25T02:36:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2023-08-25T02:36:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-08-25T02:36:37.505 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.505 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.505 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.505 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2023-08-25T02:36:37.505 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2023-08-25T02:36:37.506 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-08-25T02:36:37.507 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2023-08-25T02:36:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.508 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.509 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.509 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2023-08-25T02:36:37.509 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2023-08-25T02:36:37.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.510 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.511 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2023-08-25T02:36:37.511 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.511 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.512 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2023-08-25T02:36:37.512 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2023-08-25T02:36:37.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.512 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.513 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:37.513 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.513 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.513 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.513 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2023-08-25T02:36:37.514 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2023-08-25T02:36:37.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.515 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.515 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2023-08-25T02:36:37.515 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.516 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.516 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.516 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-08-25T02:36:37.516 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2023-08-25T02:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2023-08-25T02:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.518 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.518 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.518 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2023-08-25T02:36:37.518 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.518 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.519 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.519 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-08-25T02:36:37.519 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.519 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.519 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.520 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2023-08-25T02:36:37.520 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2023-08-25T02:36:37.520 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.520 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.521 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.521 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2023-08-25T02:36:37.521 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.521 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.521 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.522 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2023-08-25T02:36:37.523 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2023-08-25T02:36:37.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.523 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.523 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.524 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.524 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.524 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.524 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-25T02:36:37.524 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.525 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.525 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.525 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2023-08-25T02:36:37.525 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2023-08-25T02:36:37.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.526 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.526 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.526 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:37.526 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.527 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.527 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.527 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2023-08-25T02:36:37.527 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.527 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.528 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.528 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2023-08-25T02:36:37.528 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2023-08-25T02:36:37.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.529 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.529 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:37.529 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.529 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.530 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.530 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2023-08-25T02:36:37.530 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.530 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.530 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.533 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.533 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.533 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.533 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.533 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.534 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.534 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.534 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.534 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.535 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2023-08-25T02:36:37.535 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2023-08-25T02:36:37.535 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.535 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.536 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.536 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.536 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.537 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.537 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-08-25T02:36:37.537 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.537 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.537 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.538 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.538 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.538 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-08-25T02:36:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.541 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.541 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.541 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.542 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.542 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.542 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.542 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.542 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.543 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.545 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.545 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.545 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.546 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.546 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.546 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.546 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.546 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2023-08-25T02:36:37.547 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-08-25T02:36:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.549 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.549 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.549 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-08-25T02:36:37.549 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.550 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.550 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-08-25T02:36:37.550 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-08-25T02:36:37.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.551 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.551 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2023-08-25T02:36:37.551 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.551 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-08-25T02:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.553 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2023-08-25T02:36:37.553 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2023-08-25T02:36:37.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.553 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.554 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2023-08-25T02:36:37.555 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.555 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.555 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.555 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2023-08-25T02:36:37.556 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2023-08-25T02:36:37.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.556 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.556 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.557 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.557 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.557 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.557 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-08-25T02:36:37.557 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.558 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.558 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.558 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2023-08-25T02:36:37.558 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2023-08-25T02:36:37.559 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.559 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.559 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.559 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.559 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.560 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.560 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.560 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2023-08-25T02:36:37.560 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.560 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.561 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.561 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2023-08-25T02:36:37.561 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2023-08-25T02:36:37.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.562 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.563 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2023-08-25T02:36:37.563 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2023-08-25T02:36:37.563 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.563 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2023-08-25T02:36:37.563 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.564 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.564 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.564 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2023-08-25T02:36:37.564 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2023-08-25T02:36:37.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.565 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.565 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-08-25T02:36:37.565 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.566 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.566 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.566 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-08-25T02:36:37.566 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.566 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.567 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.567 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2023-08-25T02:36:37.567 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2023-08-25T02:36:37.567 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.568 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.568 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.568 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.568 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.569 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.569 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-08-25T02:36:37.569 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.569 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.569 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.570 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2023-08-25T02:36:37.570 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2023-08-25T02:36:37.570 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.570 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.571 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.571 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.571 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.571 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.571 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-08-25T02:36:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.573 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-08-25T02:36:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.575 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.575 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.575 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2023-08-25T02:36:37.575 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2023-08-25T02:36:37.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.576 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.576 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.576 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.577 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.577 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.577 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-08-25T02:36:37.577 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.577 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.578 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.578 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2023-08-25T02:36:37.578 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2023-08-25T02:36:37.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.579 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.580 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-08-25T02:36:37.580 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.580 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.580 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.581 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2023-08-25T02:36:37.581 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2023-08-25T02:36:37.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.581 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2023-08-25T02:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2023-08-25T02:36:37.583 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2023-08-25T02:36:37.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.584 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.584 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.584 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.585 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.585 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.585 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-08-25T02:36:37.585 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.585 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.586 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.586 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2023-08-25T02:36:37.586 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2023-08-25T02:36:37.586 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.587 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.587 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.587 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.588 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.588 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.588 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-08-25T02:36:37.588 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.589 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.589 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.589 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2023-08-25T02:36:37.589 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2023-08-25T02:36:37.590 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.590 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.590 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.590 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.590 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-08-25T02:36:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2023-08-25T02:36:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2023-08-25T02:36:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2023-08-25T02:36:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.595 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.595 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2023-08-25T02:36:37.595 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2023-08-25T02:36:37.596 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.596 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.596 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.596 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.596 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.597 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.597 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.597 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-08-25T02:36:37.597 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.598 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.598 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.598 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2023-08-25T02:36:37.598 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2023-08-25T02:36:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.599 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.600 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.600 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.600 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-08-25T02:36:37.600 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2023-08-25T02:36:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.604 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.604 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.604 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.604 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.604 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.605 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.605 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.605 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.605 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.605 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.606 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.606 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.606 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.607 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.608 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.608 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.608 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.608 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.609 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.610 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.610 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.610 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.610 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.611 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.611 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.611 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.611 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.611 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.612 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.612 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.612 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.612 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.612 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.613 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.613 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.613 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.614 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.614 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.614 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2023-08-25T02:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2023-08-25T02:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2023-08-25T02:36:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.618 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.618 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2023-08-25T02:36:37.618 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.618 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.618 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.619 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2023-08-25T02:36:37.619 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2023-08-25T02:36:37.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.619 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.620 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2023-08-25T02:36:37.620 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.620 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.620 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.621 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2023-08-25T02:36:37.621 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.621 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.622 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.623 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.623 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-08-25T02:36:37.623 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-08-25T02:36:37.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.623 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.624 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.624 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.624 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.624 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.624 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.625 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.625 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.625 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.625 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.626 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2023-08-25T02:36:37.626 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2023-08-25T02:36:37.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.626 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.627 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.627 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.627 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.627 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.627 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.628 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.628 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.628 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.628 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-08-25T02:36:37.629 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-08-25T02:36:37.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.629 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.629 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.630 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.630 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.630 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.631 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.631 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2023-08-25T02:36:37.631 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2023-08-25T02:36:37.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.632 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.632 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.632 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.633 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.634 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-08-25T02:36:37.634 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-08-25T02:36:37.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.634 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.635 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.635 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.635 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.635 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.636 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.636 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.636 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.636 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2023-08-25T02:36:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2023-08-25T02:36:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.638 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.638 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.638 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.638 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.638 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.639 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.639 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.639 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.639 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-08-25T02:36:37.640 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-08-25T02:36:37.640 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.640 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.640 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.640 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.642 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.642 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-08-25T02:36:37.642 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-08-25T02:36:37.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.643 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.643 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.643 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.643 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.644 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.644 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.644 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.645 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.645 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-08-25T02:36:37.645 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-08-25T02:36:37.646 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.646 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.646 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.646 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.646 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.647 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.647 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.647 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.647 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.647 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.648 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.648 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-08-25T02:36:37.648 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-08-25T02:36:37.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.651 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2023-08-25T02:36:37.651 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-08-25T02:36:37.651 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.651 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.651 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.652 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.652 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.652 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.652 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.652 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.653 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.653 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.653 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.653 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2023-08-25T02:36:37.654 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-08-25T02:36:37.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.654 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.654 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.655 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.655 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.655 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.655 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.655 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.656 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.656 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.656 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-25T02:36:37.656 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-08-25T02:36:37.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.657 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.657 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.657 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.658 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.658 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.658 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.658 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.659 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.659 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-08-25T02:36:37.659 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-08-25T02:36:37.659 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.661 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.661 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.661 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.661 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.661 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.663 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:37.663 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.663 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.663 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.663 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.664 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.664 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.664 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-08-25T02:36:37.665 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-08-25T02:36:37.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.665 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.665 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.666 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.666 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.666 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.666 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.666 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.667 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.667 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2023-08-25T02:36:37.667 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2023-08-25T02:36:37.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.668 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.668 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.668 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.669 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.670 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2023-08-25T02:36:37.670 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2023-08-25T02:36:37.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.670 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.671 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:37.671 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.671 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.671 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.672 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.672 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.672 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.672 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.673 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-08-25T02:36:37.673 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-08-25T02:36:37.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.673 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.674 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.674 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.674 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.674 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.674 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.675 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.675 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.675 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-08-25T02:36:37.676 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-08-25T02:36:37.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.676 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.676 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.678 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.678 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-08-25T02:36:37.678 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-08-25T02:36:37.678 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.679 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.679 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.679 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.679 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.680 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.680 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.680 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.680 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.681 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2023-08-25T02:36:37.681 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-08-25T02:36:37.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.681 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.682 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.683 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.683 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.683 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.683 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2023-08-25T02:36:37.684 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2023-08-25T02:36:37.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.684 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.684 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.685 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.685 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.685 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.686 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.686 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.686 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-08-25T02:36:37.686 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-08-25T02:36:37.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.687 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.687 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.687 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.688 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.688 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.688 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.688 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.689 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.689 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-08-25T02:36:37.689 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-08-25T02:36:37.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.691 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.691 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.691 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.691 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.691 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.692 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-08-25T02:36:37.692 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-08-25T02:36:37.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.692 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.693 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.694 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.694 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.694 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.694 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-08-25T02:36:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-08-25T02:36:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.695 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.696 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.696 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.696 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.696 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.696 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.697 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.697 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.697 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-08-25T02:36:37.697 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-08-25T02:36:37.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.698 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.699 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.699 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.699 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.699 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.700 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.700 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-08-25T02:36:37.700 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-08-25T02:36:37.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.701 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.701 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.701 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.702 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.702 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.702 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.702 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.702 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.703 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.704 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.704 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.704 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.705 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.705 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.705 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.705 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.705 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.706 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-08-25T02:36:37.706 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-08-25T02:36:37.706 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.707 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.707 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.707 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.707 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.707 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.708 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.708 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.708 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.709 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.709 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2023-08-25T02:36:37.709 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2023-08-25T02:36:37.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.710 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.710 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.710 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.711 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.711 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.711 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.711 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.712 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.712 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-08-25T02:36:37.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-08-25T02:36:37.713 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.713 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.713 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-08-25T02:36:37.713 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.714 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.714 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.714 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.714 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.714 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.715 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.715 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-08-25T02:36:37.715 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-08-25T02:36:37.715 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.716 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.716 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.716 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.716 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.717 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.717 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.717 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.717 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.717 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.718 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-08-25T02:36:37.718 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-08-25T02:36:37.718 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.718 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.719 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.719 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2023-08-25T02:36:37.719 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.719 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.719 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.720 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.720 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.720 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.721 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2023-08-25T02:36:37.721 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-08-25T02:36:37.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.721 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.722 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.723 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.723 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.723 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2023-08-25T02:36:37.723 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2023-08-25T02:36:37.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.724 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.724 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.724 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.725 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.725 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.725 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.725 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-08-25T02:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-08-25T02:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.727 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.728 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.728 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.728 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.729 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2023-08-25T02:36:37.729 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2023-08-25T02:36:37.729 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.729 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.729 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.730 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.730 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.730 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.730 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.730 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2023-08-25T02:36:37.731 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2023-08-25T02:36:37.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.732 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.732 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.732 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.733 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.733 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.733 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.733 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.733 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.734 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.734 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-08-25T02:36:37.734 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-08-25T02:36:37.734 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.735 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.735 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.735 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:37.735 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.735 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.736 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.736 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.736 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.736 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.736 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.737 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-08-25T02:36:37.737 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-08-25T02:36:37.737 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.737 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.738 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.738 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.738 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.738 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.740 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-08-25T02:36:37.740 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-08-25T02:36:37.740 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.740 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.740 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.741 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:37.741 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.741 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.741 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.741 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.742 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.742 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.742 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.742 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2023-08-25T02:36:37.743 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2023-08-25T02:36:37.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.743 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.743 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.743 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.744 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.744 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.744 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.744 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.744 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.745 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.745 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.745 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2023-08-25T02:36:37.745 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2023-08-25T02:36:37.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.746 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.746 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.746 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.747 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.747 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.747 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.747 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.747 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.748 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.748 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2023-08-25T02:36:37.748 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2023-08-25T02:36:37.748 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.749 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.749 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.749 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.749 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.749 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.750 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.750 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2023-08-25T02:36:37.750 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.750 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2023-08-25T02:36:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2023-08-25T02:36:37.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.752 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.752 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2023-08-25T02:36:37.752 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.753 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.753 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.753 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.753 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.753 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.754 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.754 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:37.754 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:37.755 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.755 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.755 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.755 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.755 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.756 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.756 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.756 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.756 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.757 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.757 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.757 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.758 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.758 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.758 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.759 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.759 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.759 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.759 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.760 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.760 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.760 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.760 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.761 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.761 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.761 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.761 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.761 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.762 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.762 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.762 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.763 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.763 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.763 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.763 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.764 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.765 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.765 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.765 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.765 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.766 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.766 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.766 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.766 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.766 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.767 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.767 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.767 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.767 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.767 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.768 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.768 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.768 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.768 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.769 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.770 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.770 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.770 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.770 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.770 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.771 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.771 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.771 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2023-08-25T02:36:37.771 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2023-08-25T02:36:37.772 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.772 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.772 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.772 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.773 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.773 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.773 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2023-08-25T02:36:37.773 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.774 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.775 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.775 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.775 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.775 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.775 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.776 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.776 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.776 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.777 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:37.777 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:37.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.777 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.778 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:37.778 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.778 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.778 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.778 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.779 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.779 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.779 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.779 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:37.780 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:37.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.780 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.780 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2023-08-25T02:36:37.781 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.781 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.781 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.781 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.781 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.782 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.782 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:37.782 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.784 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.784 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.784 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.784 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.785 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.785 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2023-08-25T02:36:37.785 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2023-08-25T02:36:37.785 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.786 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.786 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2023-08-25T02:36:37.786 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.786 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.787 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.787 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.787 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.787 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.787 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.788 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:37.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:37.788 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.788 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.789 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.789 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2023-08-25T02:36:37.789 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.789 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.789 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.790 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.790 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2023-08-25T02:36:37.790 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.790 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.790 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:37.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.792 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.792 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:37.792 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:37.792 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:37.793 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.793 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.793 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.793 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.794 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2023-08-25T02:36:37.794 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2023-08-25T02:36:37.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.794 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.795 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.795 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.795 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.795 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.796 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.796 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.796 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.797 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2023-08-25T02:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2023-08-25T02:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.799 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.799 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.799 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.799 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.800 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2023-08-25T02:36:37.800 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.800 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.800 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.801 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.802 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.802 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.802 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.802 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.802 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.803 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.803 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.803 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.803 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.803 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.804 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.804 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.804 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.804 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.806 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.806 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.806 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.806 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.806 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.807 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.807 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.807 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.808 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.808 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.808 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.809 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.809 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.809 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.809 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.809 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.810 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.810 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.810 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.810 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.810 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.811 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.811 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.811 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.811 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.812 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.813 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.813 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.813 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.813 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.813 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.814 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.814 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.814 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2023-08-25T02:36:37.814 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2023-08-25T02:36:37.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.815 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.815 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2023-08-25T02:36:37.815 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.816 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.816 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.816 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-08-25T02:36:37.816 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2023-08-25T02:36:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2023-08-25T02:36:37.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.818 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.818 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2023-08-25T02:36:37.818 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.818 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.819 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.819 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2023-08-25T02:36:37.819 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.820 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.820 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2023-08-25T02:36:37.820 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2023-08-25T02:36:37.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.821 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.821 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2023-08-25T02:36:37.821 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.822 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.822 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.822 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2023-08-25T02:36:37.822 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.822 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.823 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.823 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2023-08-25T02:36:37.823 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2023-08-25T02:36:37.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.824 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.824 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:37.825 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.825 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.825 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.825 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2023-08-25T02:36:37.825 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.826 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.826 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.826 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2023-08-25T02:36:37.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2023-08-25T02:36:37.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.827 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.827 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:37.828 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.828 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.828 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.828 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-08-25T02:36:37.829 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.829 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.829 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.830 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:37.830 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:37.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.830 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.832 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2023-08-25T02:36:37.832 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.832 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.833 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2023-08-25T02:36:37.833 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2023-08-25T02:36:37.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.833 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-08-25T02:36:37.834 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.835 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.835 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.836 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2023-08-25T02:36:37.836 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.836 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.836 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.837 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.837 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.837 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.839 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.839 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.839 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.839 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.839 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.840 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.840 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.840 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.840 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.841 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.841 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.841 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.842 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.842 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.842 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.842 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.843 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.844 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.844 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.844 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.845 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.845 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.845 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.845 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.845 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.846 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.846 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.846 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.846 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.846 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.847 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.847 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.847 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.847 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.848 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.848 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.849 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2023-08-25T02:36:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2023-08-25T02:36:37.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.851 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.851 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2023-08-25T02:36:37.851 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.852 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.852 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.852 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.852 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.853 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.853 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2023-08-25T02:36:37.853 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2023-08-25T02:36:37.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.855 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.855 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.855 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.855 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.856 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:37.856 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:37.856 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.856 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.857 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2023-08-25T02:36:37.857 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.857 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.857 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.857 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.858 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.858 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.858 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.858 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2023-08-25T02:36:37.859 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2023-08-25T02:36:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2023-08-25T02:36:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2023-08-25T02:36:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-08-25T02:36:37.862 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.862 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.862 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.862 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2023-08-25T02:36:37.863 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2023-08-25T02:36:37.863 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.863 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.863 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.863 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2023-08-25T02:36:37.864 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.864 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.864 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.864 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-08-25T02:36:37.864 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2023-08-25T02:36:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.866 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2023-08-25T02:36:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.867 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.868 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.868 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:37.868 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:37.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:37.869 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.869 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2023-08-25T02:36:37.869 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2023-08-25T02:36:37.869 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-08-25T02:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.871 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.871 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.871 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.872 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.872 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.872 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.872 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.872 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.873 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.874 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.874 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.874 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.874 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.874 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.875 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.875 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.875 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.876 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.876 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.876 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.876 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.876 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.878 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.878 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.878 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.879 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.879 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.879 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.879 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.879 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.880 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.880 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.880 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.880 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.880 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.881 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.881 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.881 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.881 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.881 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.882 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.883 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.883 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.883 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.883 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.883 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.884 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.884 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.884 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2023-08-25T02:36:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2023-08-25T02:36:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2023-08-25T02:36:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2023-08-25T02:36:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2023-08-25T02:36:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2023-08-25T02:36:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2023-08-25T02:36:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2023-08-25T02:36:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2023-08-25T02:36:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2023-08-25T02:36:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2023-08-25T02:36:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2023-08-25T02:36:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-08-25T02:36:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2023-08-25T02:36:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2023-08-25T02:36:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2023-08-25T02:36:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2023-08-25T02:36:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2023-08-25T02:36:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2023-08-25T02:36:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-08-25T02:36:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2023-08-25T02:36:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.904 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.904 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.904 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.904 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.904 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.907 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.907 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.907 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.907 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.907 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.908 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.908 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.908 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.908 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.909 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.909 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.909 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.909 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.909 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.910 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.910 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.910 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.910 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.910 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.912 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.912 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.912 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.912 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.913 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.913 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.913 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.913 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.913 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.914 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.914 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.914 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.914 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.914 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.915 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.915 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.915 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.915 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2023-08-25T02:36:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.930 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.930 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.930 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.930 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.930 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.931 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.931 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.931 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.931 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.932 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.932 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.932 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.933 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.933 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.933 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.933 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.933 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.934 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.934 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.935 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2023-08-25T02:36:37.935 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.936 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.936 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.936 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2023-08-25T02:36:37.936 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2023-08-25T02:36:37.936 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.937 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.937 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2023-08-25T02:36:37.937 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.937 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.938 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.938 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2023-08-25T02:36:37.938 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.938 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.938 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:37.940 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:37.940 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2023-08-25T02:36:37.940 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.940 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.941 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-08-25T02:36:37.941 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.941 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.941 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.942 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.942 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.942 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.942 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.942 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.943 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.943 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.943 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.943 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.943 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.944 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.944 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.944 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.944 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.944 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.946 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.946 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.947 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.947 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.947 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.948 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.948 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.948 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.948 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.949 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.949 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.949 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.950 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.950 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.950 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.950 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.951 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.951 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.951 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.951 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.951 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.952 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.952 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.952 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.952 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.952 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.953 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.953 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.954 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.954 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.954 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.955 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.955 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.955 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2023-08-25T02:36:37.957 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.957 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.957 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.957 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2023-08-25T02:36:37.958 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2023-08-25T02:36:37.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.958 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.958 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2023-08-25T02:36:37.959 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.959 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.959 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.959 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2023-08-25T02:36:37.959 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-08-25T02:36:37.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.960 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.960 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.960 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.961 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.962 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.962 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.962 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.962 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.962 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.963 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.963 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.963 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.963 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.963 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.964 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.964 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.964 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.964 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.965 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.965 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.965 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.966 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.967 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.967 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.967 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.967 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.968 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.968 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.968 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.968 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.968 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.969 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.969 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.969 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.969 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.970 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.970 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.970 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.970 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.970 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.971 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.971 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.971 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:37.972 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:37.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.972 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.972 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.973 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.973 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.973 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.973 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.973 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.974 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.974 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.974 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2023-08-25T02:36:37.975 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2023-08-25T02:36:37.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.975 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.975 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.975 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.976 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.976 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.976 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.976 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2023-08-25T02:36:37.976 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-08-25T02:36:37.977 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.977 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.977 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2023-08-25T02:36:37.977 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2023-08-25T02:36:37.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.978 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.978 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2023-08-25T02:36:37.978 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.979 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.980 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2023-08-25T02:36:37.980 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2023-08-25T02:36:37.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:37.980 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.981 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.981 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2023-08-25T02:36:37.981 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.981 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.981 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.982 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2023-08-25T02:36:37.982 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.982 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.982 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.983 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2023-08-25T02:36:37.983 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2023-08-25T02:36:37.983 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.983 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.983 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.984 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.984 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.984 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.984 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.984 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2023-08-25T02:36:37.985 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.985 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.985 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.985 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2023-08-25T02:36:37.986 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.987 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.987 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.987 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2023-08-25T02:36:37.987 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-08-25T02:36:37.988 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.988 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:37.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:37.989 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2023-08-25T02:36:37.989 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:37.989 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:37.989 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:37.990 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:37.990 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:37.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.990 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.991 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.992 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.992 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.992 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.992 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.992 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.993 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.993 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.993 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.993 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.994 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:37.994 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:37.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:37.994 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.994 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.995 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:37.995 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.995 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.995 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:37.995 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:37.996 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:37.996 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:37.996 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:37.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:37.996 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:37.997 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:37.997 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:37.997 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:37.997 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:37.997 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:37.998 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.000 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.000 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.000 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.001 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.001 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.001 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.001 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.002 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.002 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.002 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.002 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.002 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.003 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.003 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.004 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2023-08-25T02:36:38.004 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.004 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.004 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.005 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.005 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.005 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.005 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.005 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.006 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.006 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.006 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.006 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.006 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.007 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.009 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.009 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.009 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.010 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.010 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.010 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.010 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.010 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.011 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.011 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.011 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.011 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.012 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.013 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.013 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.013 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.013 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.013 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.015 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.015 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.015 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.016 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.016 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.016 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.017 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.017 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.017 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.017 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.017 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.018 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.018 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2023-08-25T02:36:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.020 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.020 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.020 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.021 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.022 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.022 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.022 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.022 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.022 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.023 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.023 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.023 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.023 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.023 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.024 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.024 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.024 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.024 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.026 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.026 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.026 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.026 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.027 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.027 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.027 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.027 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.027 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.028 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.028 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.028 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.028 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.028 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.029 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.029 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.029 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.029 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.029 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.030 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.030 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.030 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.030 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.031 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.031 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.031 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.032 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.032 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.032 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.032 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.033 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.033 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.033 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.033 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.033 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.034 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:38.035 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.035 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.035 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.035 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2023-08-25T02:36:38.035 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2023-08-25T02:36:38.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.036 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.036 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.037 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.037 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.037 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.037 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-08-25T02:36:38.037 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.038 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.038 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.038 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.039 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.039 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.040 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.040 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.040 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.040 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.041 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.041 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.041 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.041 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.041 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.042 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.042 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.042 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.042 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.042 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.043 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.043 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.043 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.044 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.044 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.044 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.044 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.044 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.045 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.046 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.046 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.046 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.047 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.047 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.047 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.047 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.048 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.048 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.048 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.048 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.048 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.049 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.049 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.049 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.049 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.049 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.050 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.050 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.050 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.051 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.051 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.051 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.051 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.052 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.052 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.052 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.052 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.054 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.054 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.054 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.054 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.054 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2023-08-25T02:36:38.055 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2023-08-25T02:36:38.055 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2023-08-25T02:36:38.055 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.055 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2023-08-25T02:36:38.055 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.056 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.057 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2023-08-25T02:36:38.057 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.057 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.057 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.059 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.059 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.059 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.059 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.060 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.060 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.060 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.060 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.060 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.061 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.061 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.061 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.061 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.062 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.062 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.062 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.062 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.063 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.063 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.063 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.063 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.063 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.064 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.064 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.064 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.066 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.066 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.066 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.066 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.066 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.067 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.068 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.068 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.068 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.069 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.069 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.069 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.070 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.070 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.070 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.070 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.070 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.071 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.072 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2023-08-25T02:36:38.072 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.072 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.072 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.073 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.074 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.074 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.074 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.074 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.074 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.075 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.075 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.075 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.075 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.075 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.076 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.076 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.076 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.076 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.077 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.077 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.077 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.078 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.078 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.078 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.078 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.079 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.080 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.080 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.080 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.081 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.081 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.081 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.081 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.081 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.082 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.082 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.082 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.082 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.082 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.083 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.083 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.083 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.083 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.085 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.086 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.086 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.087 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2023-08-25T02:36:38.087 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.087 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.087 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.088 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.088 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.088 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.089 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.089 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.089 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.089 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.089 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.090 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.091 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.091 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.091 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.092 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.092 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.092 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.093 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.093 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.093 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.093 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.093 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.094 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.094 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.094 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.094 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.095 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.096 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.097 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.097 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.097 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.097 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.097 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.098 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.098 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.098 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.098 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.099 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.099 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.099 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.100 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.100 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.100 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.100 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.100 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.101 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.101 INFO:teuthology.orchestra.run.smithi089.stdout: "noautoscale": { 2023-08-25T02:36:38.101 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "noautoscale", 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.102 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.103 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.103 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "global autoscale flag", 2023-08-25T02:36:38.103 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-08-25T02:36:38.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.103 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.104 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:38.104 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:38.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:38.104 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.106 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.106 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.106 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.106 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.107 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2023-08-25T02:36:38.107 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2023-08-25T02:36:38.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:38.107 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.108 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2023-08-25T02:36:38.108 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2023-08-25T02:36:38.108 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.108 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.109 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2023-08-25T02:36:38.109 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-08-25T02:36:38.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.109 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.110 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2023-08-25T02:36:38.110 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.111 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.111 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.111 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2023-08-25T02:36:38.111 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2023-08-25T02:36:38.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.112 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.112 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.112 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.112 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.112 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.113 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.113 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2023-08-25T02:36:38.113 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.113 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.113 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.115 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.115 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.115 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.115 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.116 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.116 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.116 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.116 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.117 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.117 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.117 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.117 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.118 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.119 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.119 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.119 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.119 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.119 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.120 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.120 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.120 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.121 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.121 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.121 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.121 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.122 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.122 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.122 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.122 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.125 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.126 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.126 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.126 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.126 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.126 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.127 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.127 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.127 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.127 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.128 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.128 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.128 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.128 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.129 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.129 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.129 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.129 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.129 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.130 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.130 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.130 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2023-08-25T02:36:38.131 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2023-08-25T02:36:38.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.131 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.131 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2023-08-25T02:36:38.132 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.132 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.132 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.132 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2023-08-25T02:36:38.132 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.133 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.133 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.133 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2023-08-25T02:36:38.133 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2023-08-25T02:36:38.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:38.134 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.134 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.134 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2023-08-25T02:36:38.134 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.135 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.136 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2023-08-25T02:36:38.137 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.137 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.137 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.137 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2023-08-25T02:36:38.138 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2023-08-25T02:36:38.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.138 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.138 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.140 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.140 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2023-08-25T02:36:38.140 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2023-08-25T02:36:38.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.142 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.142 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-08-25T02:36:38.142 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-08-25T02:36:38.142 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.142 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.143 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.143 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.143 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.144 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.144 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.144 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.144 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.145 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.145 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.145 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.145 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.145 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.146 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.147 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.147 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.147 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.148 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.148 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.148 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.148 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.148 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.149 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.149 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.149 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.149 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.150 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.151 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.151 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.151 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.151 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.151 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.152 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.152 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.152 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.152 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.152 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.153 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.153 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.153 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.154 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.154 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.154 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.155 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.156 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.156 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.157 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.158 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.158 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.158 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.158 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.158 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2023-08-25T02:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-08-25T02:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.160 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.160 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.160 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2023-08-25T02:36:38.160 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.160 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2023-08-25T02:36:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2023-08-25T02:36:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.163 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2023-08-25T02:36:38.163 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.163 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.163 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.164 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:38.165 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:38.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.165 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.165 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2023-08-25T02:36:38.166 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.166 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.166 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.166 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-08-25T02:36:38.166 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.167 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.167 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.167 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:38.168 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:38.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.168 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.168 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2023-08-25T02:36:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2023-08-25T02:36:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2023-08-25T02:36:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2023-08-25T02:36:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2023-08-25T02:36:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2023-08-25T02:36:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2023-08-25T02:36:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2023-08-25T02:36:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2023-08-25T02:36:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2023-08-25T02:36:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2023-08-25T02:36:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2023-08-25T02:36:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.182 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.183 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.183 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.183 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.183 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.183 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.184 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.184 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.184 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.185 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.185 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.185 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.185 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.185 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.186 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.186 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.186 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.186 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.186 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.187 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.187 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.187 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.189 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.189 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.189 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.189 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.189 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.190 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.190 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.190 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.190 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.190 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.191 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.191 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.191 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.192 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.192 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.192 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.192 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.193 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.193 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.193 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.193 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.193 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.195 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2023-08-25T02:36:38.195 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.195 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.195 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.195 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.196 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.196 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.196 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2023-08-25T02:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2023-08-25T02:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.198 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.198 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.198 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.198 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.198 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.199 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.199 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2023-08-25T02:36:38.200 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2023-08-25T02:36:38.200 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.200 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.200 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.201 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.201 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.201 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.201 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.201 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.202 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.203 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2023-08-25T02:36:38.203 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.203 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.203 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.204 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2023-08-25T02:36:38.204 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2023-08-25T02:36:38.204 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.204 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.205 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.205 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.205 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.205 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.205 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.206 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.206 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.206 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.207 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.208 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.208 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.208 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.208 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.209 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.209 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.209 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.210 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.210 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.210 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.210 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.211 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.211 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.211 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.211 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.211 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.212 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.212 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.212 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.212 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.212 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.213 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.213 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.213 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.213 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.214 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.215 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.216 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.216 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.216 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.217 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.217 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.217 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.217 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.218 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.219 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.219 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.219 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.219 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.219 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.220 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.220 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.220 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.221 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.221 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.221 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.221 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.222 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.222 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.222 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.223 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2023-08-25T02:36:38.223 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2023-08-25T02:36:38.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.223 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.224 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.225 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.225 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2023-08-25T02:36:38.225 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2023-08-25T02:36:38.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.226 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.226 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.226 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.227 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.227 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.227 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.227 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.227 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.228 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.229 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.230 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.231 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.231 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.231 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.231 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.231 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.233 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.234 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.234 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.234 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.234 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.235 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.235 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.235 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.235 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.235 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.236 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.236 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.236 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.237 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.237 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.237 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.237 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.238 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.239 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.239 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.239 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.239 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.239 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.240 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.240 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.240 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.241 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.241 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.241 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.241 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.241 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.242 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.242 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.242 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.242 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2023-08-25T02:36:38.243 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.244 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.244 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.244 INFO:teuthology.orchestra.run.smithi089.stdout: "drive_group_interval": { 2023-08-25T02:36:38.244 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "drive_group_interval", 2023-08-25T02:36:38.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:38.245 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.245 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.245 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300.0", 2023-08-25T02:36:38.245 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.245 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.246 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.246 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-08-25T02:36:38.246 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.246 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.246 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.247 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.247 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.247 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.248 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.248 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.248 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.248 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.248 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.249 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.249 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.249 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.249 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.249 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.250 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.250 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.250 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.250 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.251 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.251 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.251 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.252 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.252 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.252 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.252 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.255 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.255 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.255 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.255 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.255 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.256 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.256 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.256 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.256 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.256 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.257 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.257 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.257 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.257 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.257 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.258 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.258 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.258 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.259 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.259 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.259 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.259 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.260 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.260 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.260 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.261 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2023-08-25T02:36:38.261 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2023-08-25T02:36:38.261 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.261 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.261 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-08-25T02:36:38.262 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.263 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.264 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2023-08-25T02:36:38.264 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.264 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.264 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.265 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.265 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.265 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.265 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.266 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.266 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.266 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.266 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.266 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.267 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.269 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.269 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.269 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.270 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.270 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.270 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.270 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.271 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.273 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.273 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.273 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.273 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.273 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.276 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.276 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.277 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.278 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.278 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2023-08-25T02:36:38.278 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2023-08-25T02:36:38.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.279 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.279 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.279 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.280 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.280 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.280 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.280 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.281 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.281 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2023-08-25T02:36:38.281 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2023-08-25T02:36:38.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.282 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.282 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2023-08-25T02:36:38.282 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.283 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.283 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.283 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.283 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.283 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.284 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.284 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2023-08-25T02:36:38.284 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2023-08-25T02:36:38.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.285 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.285 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.285 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.285 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.287 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2023-08-25T02:36:38.287 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2023-08-25T02:36:38.287 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.287 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.288 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.288 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.288 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.288 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.288 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.289 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.289 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.289 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.289 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2023-08-25T02:36:38.290 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2023-08-25T02:36:38.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2023-08-25T02:36:38.290 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.290 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.292 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.292 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2023-08-25T02:36:38.292 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2023-08-25T02:36:38.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.293 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.293 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.293 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.293 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.294 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.294 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.294 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.294 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.294 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.295 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2023-08-25T02:36:38.295 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2023-08-25T02:36:38.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.295 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.296 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.296 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.296 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.296 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.297 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.297 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.297 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.297 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.298 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.299 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.299 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.299 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.299 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.300 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.300 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.300 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2023-08-25T02:36:38.300 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2023-08-25T02:36:38.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.301 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.301 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.301 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2023-08-25T02:36:38.302 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2023-08-25T02:36:38.302 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.302 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.302 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.303 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.304 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.304 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.304 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.304 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.305 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.305 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.305 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.306 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2023-08-25T02:36:38.306 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2023-08-25T02:36:38.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.306 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.307 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.307 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.307 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.307 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.307 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.308 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.308 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.308 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2023-08-25T02:36:38.309 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2023-08-25T02:36:38.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.309 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.309 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2023-08-25T02:36:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.312 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2023-08-25T02:36:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2023-08-25T02:36:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.313 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.314 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2023-08-25T02:36:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2023-08-25T02:36:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2023-08-25T02:36:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.316 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.317 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.318 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.319 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.321 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2023-08-25T02:36:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.337 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.337 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.337 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.337 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.337 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.338 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.339 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.339 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.339 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.339 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.340 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.340 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.340 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.340 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.340 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.341 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.341 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.341 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.341 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.341 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.342 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.342 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.342 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.343 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.343 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.343 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.343 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.344 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.344 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.344 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.344 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.344 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.345 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.345 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.345 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.345 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.345 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.346 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.347 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.347 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.347 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.347 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.347 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.348 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.348 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.348 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.348 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.349 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2023-08-25T02:36:38.350 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.350 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.350 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.350 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.350 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.351 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.351 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.351 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.351 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.352 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.352 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.352 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.352 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.352 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.353 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.354 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.354 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.354 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.354 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.355 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.355 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.355 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.355 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.356 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.356 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.356 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.356 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.361 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.361 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.361 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.362 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.362 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.362 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.362 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.363 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.363 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.363 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.363 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.364 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.365 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2023-08-25T02:36:38.365 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.365 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.366 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.366 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2023-08-25T02:36:38.366 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2023-08-25T02:36:38.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.366 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.367 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.368 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.368 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.368 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.368 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:38.369 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:38.369 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:38.369 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.369 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.369 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2023-08-25T02:36:38.370 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.370 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.370 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.370 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.370 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.371 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.371 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.371 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.371 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.372 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.372 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.372 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.372 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.373 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.374 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.374 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.374 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.374 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.374 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.375 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.375 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.375 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.375 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.376 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.376 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.376 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.377 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.377 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.377 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.377 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.378 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.378 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.378 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.378 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.379 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.379 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.379 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.379 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.381 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.381 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.381 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.381 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.381 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.382 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.382 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.382 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.383 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.383 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.383 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.383 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.383 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.385 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.385 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.385 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.386 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2023-08-25T02:36:38.386 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.387 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.387 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.387 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2023-08-25T02:36:38.387 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2023-08-25T02:36:38.387 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.388 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.388 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.388 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.388 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.388 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.389 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.389 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2023-08-25T02:36:38.389 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.389 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.389 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2023-08-25T02:36:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2023-08-25T02:36:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.390 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.391 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.391 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.391 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.391 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.391 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.392 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-08-25T02:36:38.392 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.392 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.392 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.393 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2023-08-25T02:36:38.393 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2023-08-25T02:36:38.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.393 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-08-25T02:36:38.394 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.395 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.395 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.395 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2023-08-25T02:36:38.395 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2023-08-25T02:36:38.396 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.396 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.396 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.396 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.396 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.397 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.397 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.397 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-08-25T02:36:38.397 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.398 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.398 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.398 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2023-08-25T02:36:38.399 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2023-08-25T02:36:38.399 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.399 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.399 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.400 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.400 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.400 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.400 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.401 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2023-08-25T02:36:38.401 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.401 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.401 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.402 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2023-08-25T02:36:38.402 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2023-08-25T02:36:38.402 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.403 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.403 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.403 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.404 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.404 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.404 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.404 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.405 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.405 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.405 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2023-08-25T02:36:38.406 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2023-08-25T02:36:38.406 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.406 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.407 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.407 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.407 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.407 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.407 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.408 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.408 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.409 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.409 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2023-08-25T02:36:38.409 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2023-08-25T02:36:38.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.410 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.410 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-08-25T02:36:38.411 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.411 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.411 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.411 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.412 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.412 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.412 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.413 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2023-08-25T02:36:38.413 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2023-08-25T02:36:38.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.413 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.414 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.414 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.414 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.415 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.415 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.415 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.415 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.416 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:38.416 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:38.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.417 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.417 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2023-08-25T02:36:38.417 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2023-08-25T02:36:38.417 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.419 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2023-08-25T02:36:38.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2023-08-25T02:36:38.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.419 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.419 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.420 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2023-08-25T02:36:38.420 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.420 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.420 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.420 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.421 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.421 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.421 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.421 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2023-08-25T02:36:38.422 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2023-08-25T02:36:38.422 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.422 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.422 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.423 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.423 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.423 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.423 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.423 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.424 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.424 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.425 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2023-08-25T02:36:38.425 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2023-08-25T02:36:38.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.425 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.425 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.426 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.426 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.426 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.426 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.426 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.427 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.427 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.427 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.427 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.428 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.428 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.429 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.429 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.429 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.430 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.431 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.431 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.431 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.431 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.432 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.432 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.432 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.432 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.433 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.433 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.433 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.434 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.434 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.434 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.434 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.434 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.435 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.435 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.435 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.436 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.436 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.436 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.437 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.437 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.437 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.437 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.437 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.438 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.439 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.439 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.439 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.440 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.440 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.441 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.441 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.441 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.441 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.443 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2023-08-25T02:36:38.443 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2023-08-25T02:36:38.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.443 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.444 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.444 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.444 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.444 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.444 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.445 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.445 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.445 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.445 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.446 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2023-08-25T02:36:38.446 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2023-08-25T02:36:38.446 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.446 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.446 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.447 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.447 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.447 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.447 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.447 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.448 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.448 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.448 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.448 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-08-25T02:36:38.449 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.450 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.450 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.450 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.450 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.450 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.451 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.452 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2023-08-25T02:36:38.452 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.452 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.452 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.452 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.454 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.454 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.454 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.454 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.454 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.455 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.455 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.455 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.455 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.455 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.456 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.456 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.456 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.456 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.457 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.458 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.458 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.458 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.458 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.458 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.459 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.459 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.459 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.460 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.460 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.461 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.461 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.461 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.461 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.461 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.462 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.462 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.462 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.462 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.462 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.463 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.466 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.467 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2023-08-25T02:36:38.467 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.467 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.467 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.467 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.468 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.468 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.468 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.468 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.470 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.471 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.471 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.471 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.471 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.472 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.472 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.472 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.473 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.473 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.473 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.473 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.473 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.474 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.474 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.474 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.474 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.475 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.476 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.476 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.476 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.476 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.476 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.477 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.477 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.477 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.477 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.477 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.478 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.478 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.478 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.478 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.479 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.479 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.479 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.479 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.479 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.480 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.480 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.480 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.480 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.480 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.481 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.482 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.482 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2023-08-25T02:36:38.482 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.482 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.482 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.483 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2023-08-25T02:36:38.483 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.483 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.484 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2023-08-25T02:36:38.484 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2023-08-25T02:36:38.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.484 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.484 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.485 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2023-08-25T02:36:38.485 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.485 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.485 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.485 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-08-25T02:36:38.486 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.486 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.486 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.486 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.486 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.487 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2023-08-25T02:36:38.487 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2023-08-25T02:36:38.487 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.489 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2023-08-25T02:36:38.489 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.489 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.489 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.489 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.490 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.490 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.490 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2023-08-25T02:36:38.491 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2023-08-25T02:36:38.491 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.491 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.491 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.491 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.492 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.492 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.492 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.492 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.492 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.493 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.493 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.493 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2023-08-25T02:36:38.493 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2023-08-25T02:36:38.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2023-08-25T02:36:38.494 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.494 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2023-08-25T02:36:38.494 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.496 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.496 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2023-08-25T02:36:38.496 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2023-08-25T02:36:38.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.498 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.499 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.499 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.499 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.499 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.500 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.500 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2023-08-25T02:36:38.500 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2023-08-25T02:36:38.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.501 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.501 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.501 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.501 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.502 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.502 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.502 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.502 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.502 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.503 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2023-08-25T02:36:38.503 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2023-08-25T02:36:38.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.503 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.504 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2023-08-25T02:36:38.504 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.504 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.504 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2023-08-25T02:36:38.505 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2023-08-25T02:36:38.506 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.506 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.506 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.506 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.506 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.507 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2023-08-25T02:36:38.507 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2023-08-25T02:36:38.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2023-08-25T02:36:38.507 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.508 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2023-08-25T02:36:38.508 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2023-08-25T02:36:38.508 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.508 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.508 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.509 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.509 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.509 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.509 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.510 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2023-08-25T02:36:38.510 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2023-08-25T02:36:38.510 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.510 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2023-08-25T02:36:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.512 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.512 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.512 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.512 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2023-08-25T02:36:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2023-08-25T02:36:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2023-08-25T02:36:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2023-08-25T02:36:38.514 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.514 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.514 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.514 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.514 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.515 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.515 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2023-08-25T02:36:38.515 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2023-08-25T02:36:38.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2023-08-25T02:36:38.516 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2023-08-25T02:36:38.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2023-08-25T02:36:38.516 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-08-25T02:36:38.516 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2023-08-25T02:36:38.517 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2023-08-25T02:36:38.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.518 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.518 INFO:teuthology.orchestra.run.smithi089.stdout: "services": { 2023-08-25T02:36:38.519 INFO:teuthology.orchestra.run.smithi089.stdout: "dashboard": "https://172.21.15.89:8443/" 2023-08-25T02:36:38.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.519 INFO:teuthology.orchestra.run.smithi089.stdout: "always_on_modules": { 2023-08-25T02:36:38.519 INFO:teuthology.orchestra.run.smithi089.stdout: "octopus": [ 2023-08-25T02:36:38.519 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2023-08-25T02:36:38.520 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2023-08-25T02:36:38.520 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2023-08-25T02:36:38.520 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2023-08-25T02:36:38.520 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2023-08-25T02:36:38.520 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.521 INFO:teuthology.orchestra.run.smithi089.stdout: "pacific": [ 2023-08-25T02:36:38.522 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2023-08-25T02:36:38.522 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2023-08-25T02:36:38.522 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2023-08-25T02:36:38.522 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2023-08-25T02:36:38.522 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2023-08-25T02:36:38.523 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2023-08-25T02:36:38.523 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2023-08-25T02:36:38.523 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2023-08-25T02:36:38.523 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2023-08-25T02:36:38.523 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2023-08-25T02:36:38.524 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.524 INFO:teuthology.orchestra.run.smithi089.stdout: "quincy": [ 2023-08-25T02:36:38.524 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2023-08-25T02:36:38.524 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2023-08-25T02:36:38.524 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2023-08-25T02:36:38.525 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2023-08-25T02:36:38.526 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2023-08-25T02:36:38.526 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-08-25T02:36:38.526 INFO:teuthology.orchestra.run.smithi089.stdout: "reef": [ 2023-08-25T02:36:38.526 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2023-08-25T02:36:38.526 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2023-08-25T02:36:38.527 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2023-08-25T02:36:38.527 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2023-08-25T02:36:38.527 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2023-08-25T02:36:38.527 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2023-08-25T02:36:38.527 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2023-08-25T02:36:38.528 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2023-08-25T02:36:38.528 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2023-08-25T02:36:38.528 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2023-08-25T02:36:38.528 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: "last_failure_osd_epoch": 83, 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: "active_clients": [ 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 87081324 2023-08-25T02:36:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.530 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.531 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "libcephsqlite", 2023-08-25T02:36:38.531 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.532 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.532 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 701372317 2023-08-25T02:36:38.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.532 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.533 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2023-08-25T02:36:38.533 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.534 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.534 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 2237146277 2023-08-25T02:36:38.534 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.534 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.535 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.536 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.536 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3984506757 2023-08-25T02:36:38.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.537 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2023-08-25T02:36:38.537 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.537 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.538 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.538 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 717470616 2023-08-25T02:36:38.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.538 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:38.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.539 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2023-08-25T02:36:38.539 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2023-08-25T02:36:38.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-08-25T02:36:38.539 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2023-08-25T02:36:38.540 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2023-08-25T02:36:38.540 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 2463400436 2023-08-25T02:36:38.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:38.541 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-08-25T02:36:38.541 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:36:38.543 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:34 smithi122 ceph-mon[100457]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:38.543 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/4098694009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.543 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/867576281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.543 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:34 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/202621427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.544 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:35 smithi122 ceph-mon[100457]: pgmap v313: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:38.544 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:35 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/285867118' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-08-25T02:36:38.544 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:37 smithi122 ceph-mon[100457]: pgmap v314: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:38.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:34 smithi121 ceph-mon[100099]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-08-25T02:36:38.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/4098694009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/867576281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:34 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/202621427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:36:38.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:35 smithi121 ceph-mon[100099]: pgmap v313: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:38.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:35 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/285867118' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-08-25T02:36:38.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:37 smithi121 ceph-mon[100099]: pgmap v314: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:38.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 msgr2=0x7ffa700a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:38.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 0x7ffa700a25c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009c60 tx=0x7ffa74009260 comp rx=0 tx=0).stop 2023-08-25T02:36:38.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 msgr2=0x7ffa8419b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:36:38.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00ef30 tx=0x7ffa7c00c3a0 comp rx=0 tx=0).stop 2023-08-25T02:36:38.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 shutdown_connections 2023-08-25T02:36:38.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7ffa700a0100 0x7ffa700a25c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:38.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa841004d0 0x7ffa84196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:38.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa8419b330 0x7ffa8419b7e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:38.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 --2- 172.21.15.89:0/285867118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa84104b00 0x7ffa84196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:36:38.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 >> 172.21.15.89:0/285867118 conn(0x7ffa84074a00 msgr2=0x7ffa84103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:36:38.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 shutdown_connections 2023-08-25T02:36:38.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:36:34.876+0000 7ffa8a041700 1 -- 172.21.15.89:0/285867118 wait complete. 2023-08-25T02:36:38.617 DEBUG:tasks.check_counter:Getting stats from a 2023-08-25T02:36:38.618 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.a perf dump 2023-08-25T02:36:38.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:35 smithi089 ceph-mon[98732]: pgmap v313: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:38.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:35 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/285867118' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-08-25T02:36:38.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:37 smithi089 ceph-mon[98732]: pgmap v314: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:39.181 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:36:39.694 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:39 smithi089 ceph-mon[98732]: pgmap v315: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:39.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:39 smithi121 ceph-mon[100099]: pgmap v315: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:39.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:39 smithi122 ceph-mon[100457]: pgmap v315: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:40.414 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:36:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4, 2023-08-25T02:36:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 4, 2023-08-25T02:36:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 2992, 2023-08-25T02:36:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 448, 2023-08-25T02:36:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2023-08-25T02:36:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2023-08-25T02:36:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.001652822, 2023-08-25T02:36:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000032761, 2023-08-25T02:36:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000606911, 2023-08-25T02:36:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3, 2023-08-25T02:36:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000697095, 2023-08-25T02:36:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000232365 2023-08-25T02:36:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-08-25T02:36:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 448 2023-08-25T02:36:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2, 2023-08-25T02:36:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 145, 2023-08-25T02:36:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 320, 2023-08-25T02:36:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 177952, 2023-08-25T02:36:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2023-08-25T02:36:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.012684260, 2023-08-25T02:36:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.008411784, 2023-08-25T02:36:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.001920696, 2023-08-25T02:36:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 142, 2023-08-25T02:36:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.014730202, 2023-08-25T02:36:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000103733 2023-08-25T02:36:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 320, 2023-08-25T02:36:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 177952 2023-08-25T02:36:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 197, 2023-08-25T02:36:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 189, 2023-08-25T02:36:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1134832, 2023-08-25T02:36:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 113200, 2023-08-25T02:36:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2023-08-25T02:36:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.036286185, 2023-08-25T02:36:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.012936715, 2023-08-25T02:36:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.023856233, 2023-08-25T02:36:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004438020, 2023-08-25T02:36:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 188, 2023-08-25T02:36:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.008985945, 2023-08-25T02:36:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000047797 2023-08-25T02:36:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1134832, 2023-08-25T02:36:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 113200 2023-08-25T02:36:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2023-08-25T02:36:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2023-08-25T02:36:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2023-08-25T02:36:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2023-08-25T02:36:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2023-08-25T02:36:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 149017, 2023-08-25T02:36:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 31, 2023-08-25T02:36:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2023-08-25T02:36:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2023-08-25T02:36:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2023-08-25T02:36:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2023-08-25T02:36:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2023-08-25T02:36:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2023-08-25T02:36:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2023-08-25T02:36:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2023-08-25T02:36:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2023-08-25T02:36:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2023-08-25T02:36:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 199, 2023-08-25T02:36:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1107050, 2023-08-25T02:36:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 199, 2023-08-25T02:36:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2023-08-25T02:36:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2023-08-25T02:36:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 199, 2023-08-25T02:36:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1107050, 2023-08-25T02:36:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2023-08-25T02:36:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:40.455 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:36:40.841 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-08-25T02:36:40.842 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-08-25T02:36:40.843 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-08-25T02:36:40.843 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-08-25T02:36:40.844 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-08-25T02:36:40.844 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-08-25T02:36:40.845 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-08-25T02:36:40.845 DEBUG:tasks.check_counter:Getting stats from d 2023-08-25T02:36:40.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.d perf dump 2023-08-25T02:36:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:36:41.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:41 smithi089 ceph-mon[98732]: pgmap v316: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:41.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:41 smithi121 ceph-mon[100099]: pgmap v316: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:41.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:41 smithi122 ceph-mon[100457]: pgmap v316: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:42.640 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:36:42.641 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:42.641 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 0, 2023-08-25T02:36:42.641 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2, 2023-08-25T02:36:42.641 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 0, 2023-08-25T02:36:42.642 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 0, 2023-08-25T02:36:42.642 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2023-08-25T02:36:42.642 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2023-08-25T02:36:42.642 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.001172027, 2023-08-25T02:36:42.642 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000000000, 2023-08-25T02:36:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000216939, 2023-08-25T02:36:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:36:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:36:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 203, 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 195, 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1137088, 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 116656, 2023-08-25T02:36:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2023-08-25T02:36:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.037796083, 2023-08-25T02:36:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.013054512, 2023-08-25T02:36:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.025227037, 2023-08-25T02:36:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004923694, 2023-08-25T02:36:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 194, 2023-08-25T02:36:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010451370, 2023-08-25T02:36:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000053873 2023-08-25T02:36:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1137088, 2023-08-25T02:36:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 116656 2023-08-25T02:36:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 6, 2023-08-25T02:36:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 151, 2023-08-25T02:36:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 3312, 2023-08-25T02:36:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 179104, 2023-08-25T02:36:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2023-08-25T02:36:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.012072599, 2023-08-25T02:36:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.007800175, 2023-08-25T02:36:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.002039685, 2023-08-25T02:36:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 149, 2023-08-25T02:36:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.017936971, 2023-08-25T02:36:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000120382 2023-08-25T02:36:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-08-25T02:36:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 179104 2023-08-25T02:36:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2023-08-25T02:36:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2023-08-25T02:36:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2023-08-25T02:36:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2023-08-25T02:36:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2023-08-25T02:36:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148994, 2023-08-25T02:36:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 31, 2023-08-25T02:36:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2023-08-25T02:36:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2023-08-25T02:36:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2023-08-25T02:36:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2023-08-25T02:36:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2023-08-25T02:36:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2023-08-25T02:36:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2023-08-25T02:36:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2023-08-25T02:36:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2023-08-25T02:36:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2023-08-25T02:36:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2023-08-25T02:36:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 205, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1108467, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 205, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2023-08-25T02:36:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2023-08-25T02:36:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 205, 2023-08-25T02:36:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1108467, 2023-08-25T02:36:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2023-08-25T02:36:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:42.672 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:36:42.962 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-08-25T02:36:42.962 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-08-25T02:36:42.962 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-08-25T02:36:42.963 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-08-25T02:36:42.963 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-08-25T02:36:42.963 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-08-25T02:36:42.964 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-08-25T02:36:42.964 DEBUG:tasks.check_counter:Getting stats from g 2023-08-25T02:36:42.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.g perf dump 2023-08-25T02:36:43.517 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:36:43.876 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:43 smithi089 ceph-mon[98732]: pgmap v317: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:43.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:43 smithi121 ceph-mon[100099]: pgmap v317: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:43.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:43 smithi122 ceph-mon[100457]: pgmap v317: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:44.727 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:36:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 199, 2023-08-25T02:36:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 191, 2023-08-25T02:36:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1135376, 2023-08-25T02:36:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 114448, 2023-08-25T02:36:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2023-08-25T02:36:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2023-08-25T02:36:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.037110736, 2023-08-25T02:36:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.012571620, 2023-08-25T02:36:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.024684567, 2023-08-25T02:36:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004620041, 2023-08-25T02:36:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 190, 2023-08-25T02:36:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010575546, 2023-08-25T02:36:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000055660 2023-08-25T02:36:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1135376, 2023-08-25T02:36:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 114448 2023-08-25T02:36:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 6, 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 148, 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 3312, 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 178576, 2023-08-25T02:36:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2023-08-25T02:36:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.012359531, 2023-08-25T02:36:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.007977039, 2023-08-25T02:36:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.002210225, 2023-08-25T02:36:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 146, 2023-08-25T02:36:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.012179563, 2023-08-25T02:36:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000083421 2023-08-25T02:36:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-08-25T02:36:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 178576 2023-08-25T02:36:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 0, 2023-08-25T02:36:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2, 2023-08-25T02:36:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 0, 2023-08-25T02:36:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 0, 2023-08-25T02:36:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2023-08-25T02:36:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.000363288, 2023-08-25T02:36:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000000000, 2023-08-25T02:36:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000269805, 2023-08-25T02:36:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:36:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:36:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2023-08-25T02:36:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:44.746 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:44.746 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:44.746 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:44.746 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:44.747 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:44.747 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:44.747 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:44.748 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:44.748 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:44.748 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:44.749 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:44.749 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:44.749 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:44.749 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:44.750 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:44.750 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:44.750 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:44.750 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:44.751 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:44.751 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:44.751 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:44.751 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:44.751 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:44.752 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:44.752 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:44.752 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:44.752 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:44.753 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:44.753 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2023-08-25T02:36:44.753 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:44.753 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:44.753 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:44.754 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:44.754 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148977, 2023-08-25T02:36:44.754 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 30, 2023-08-25T02:36:44.754 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:44.754 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:44.755 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:44.756 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2023-08-25T02:36:44.756 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2023-08-25T02:36:44.756 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:44.756 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:44.757 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:44.757 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2023-08-25T02:36:44.757 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2023-08-25T02:36:44.757 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2023-08-25T02:36:44.757 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:44.758 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2023-08-25T02:36:44.758 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:44.758 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2023-08-25T02:36:44.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:44.758 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:44.759 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2023-08-25T02:36:44.759 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2023-08-25T02:36:44.759 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2023-08-25T02:36:44.760 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 201, 2023-08-25T02:36:44.760 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1107310, 2023-08-25T02:36:44.760 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:44.761 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 201, 2023-08-25T02:36:44.761 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2023-08-25T02:36:44.761 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2023-08-25T02:36:44.762 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 201, 2023-08-25T02:36:44.762 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1107310, 2023-08-25T02:36:44.762 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2023-08-25T02:36:44.762 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:44.763 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:44.763 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:44.763 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:44.764 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:44.764 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:36:45.176 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-08-25T02:36:45.177 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-08-25T02:36:45.177 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-08-25T02:36:45.177 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-08-25T02:36:45.177 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-08-25T02:36:45.178 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-08-25T02:36:45.178 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-08-25T02:36:45.178 DEBUG:tasks.check_counter:Getting stats from j 2023-08-25T02:36:45.179 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.j perf dump 2023-08-25T02:36:45.732 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.a/config 2023-08-25T02:36:45.949 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:45 smithi089 ceph-mon[98732]: pgmap v318: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:45.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:45 smithi121 ceph-mon[100099]: pgmap v318: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:45.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:45 smithi122 ceph-mon[100457]: pgmap v318: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-08-25T02:36:46.968 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-08-25T02:36:46.969 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:46.969 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4, 2023-08-25T02:36:46.969 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 4, 2023-08-25T02:36:46.969 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 2992, 2023-08-25T02:36:46.970 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 448, 2023-08-25T02:36:46.970 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2023-08-25T02:36:46.970 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2023-08-25T02:36:46.970 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.001569318, 2023-08-25T02:36:46.970 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000035297, 2023-08-25T02:36:46.971 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000693742, 2023-08-25T02:36:46.971 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:46.971 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:46.971 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3, 2023-08-25T02:36:46.971 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000826585, 2023-08-25T02:36:46.972 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000275528 2023-08-25T02:36:46.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.972 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:46.972 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:46.972 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-08-25T02:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 448 2023-08-25T02:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.974 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:46.974 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 204, 2023-08-25T02:36:46.974 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 196, 2023-08-25T02:36:46.974 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1137360, 2023-08-25T02:36:46.974 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 117280, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.038407219, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.014082032, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.024995800, 2023-08-25T02:36:46.975 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004807824, 2023-08-25T02:36:46.976 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:46.976 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 195, 2023-08-25T02:36:46.976 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010657772, 2023-08-25T02:36:46.976 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000054655 2023-08-25T02:36:46.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.977 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:46.977 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:46.977 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:46.977 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:46.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.978 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1137360, 2023-08-25T02:36:46.978 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 117280 2023-08-25T02:36:46.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.978 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:46.979 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2, 2023-08-25T02:36:46.979 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 150, 2023-08-25T02:36:46.979 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 320, 2023-08-25T02:36:46.979 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 178832, 2023-08-25T02:36:46.979 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:46.980 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2023-08-25T02:36:46.980 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.013144560, 2023-08-25T02:36:46.980 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.009072129, 2023-08-25T02:36:46.980 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.001727201, 2023-08-25T02:36:46.980 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:46.981 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:46.981 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 147, 2023-08-25T02:36:46.981 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.014903174, 2023-08-25T02:36:46.981 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000101382 2023-08-25T02:36:46.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.982 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 320, 2023-08-25T02:36:46.983 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 178832 2023-08-25T02:36:46.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.983 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2023-08-25T02:36:46.983 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2023-08-25T02:36:46.983 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2023-08-25T02:36:46.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.984 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2023-08-25T02:36:46.984 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:46.984 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:46.985 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:46.986 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:46.986 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:46.986 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:46.986 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:46.987 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:46.988 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:46.988 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:46.988 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:46.988 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:46.989 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:46.989 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:46.989 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:46.989 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:46.989 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:46.990 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:46.990 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:46.990 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:46.990 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:46.990 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2023-08-25T02:36:46.991 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:46.991 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:46.991 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:46.991 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:46.991 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 149057, 2023-08-25T02:36:46.992 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 32, 2023-08-25T02:36:46.992 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:46.992 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:46.992 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2023-08-25T02:36:46.993 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2023-08-25T02:36:46.993 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:46.993 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:46.993 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:46.994 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:46.994 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2023-08-25T02:36:46.994 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2023-08-25T02:36:46.994 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:46.994 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:46.995 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:46.995 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2023-08-25T02:36:46.995 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2023-08-25T02:36:46.995 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2023-08-25T02:36:46.995 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:46.996 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2023-08-25T02:36:46.996 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:46.996 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2023-08-25T02:36:46.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-08-25T02:36:46.996 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2023-08-25T02:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2023-08-25T02:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2023-08-25T02:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 206, 2023-08-25T02:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1108597, 2023-08-25T02:36:46.998 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:46.998 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 206, 2023-08-25T02:36:46.998 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2023-08-25T02:36:46.998 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2023-08-25T02:36:46.998 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 206, 2023-08-25T02:36:46.999 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1108597, 2023-08-25T02:36:46.999 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2023-08-25T02:36:46.999 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2023-08-25T02:36:46.999 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2023-08-25T02:36:46.999 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2023-08-25T02:36:47.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:47.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-08-25T02:36:47.000 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-08-25T02:36:47.396 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-08-25T02:36:47.396 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-08-25T02:36:47.397 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-08-25T02:36:47.397 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-08-25T02:36:47.397 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-08-25T02:36:47.398 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-08-25T02:36:47.398 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-08-25T02:36:47.398 DEBUG:tasks.check_counter:Getting stats from b 2023-08-25T02:36:47.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.b perf dump 2023-08-25T02:36:47.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:47 smithi121 ceph-mon[100099]: pgmap v319: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:47.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:47 smithi089 ceph-mon[98732]: pgmap v319: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:47.969 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:36:47.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:47 smithi122 ceph-mon[100457]: pgmap v319: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-08-25T02:36:49.321 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-08-25T02:36:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 5784, 2023-08-25T02:36:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 5177, 2023-08-25T02:36:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 4181363, 2023-08-25T02:36:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 7332134, 2023-08-25T02:36:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 13, 2023-08-25T02:36:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2023-08-25T02:36:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.461046806, 2023-08-25T02:36:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.177480793, 2023-08-25T02:36:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.230922681, 2023-08-25T02:36:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.080932481, 2023-08-25T02:36:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 5176, 2023-08-25T02:36:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 5.871778851, 2023-08-25T02:36:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001134424 2023-08-25T02:36:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 8026, 2023-08-25T02:36:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.001495125, 2023-08-25T02:36:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000186 2023-08-25T02:36:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1817680, 2023-08-25T02:36:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 262864 2023-08-25T02:36:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 8616, 2023-08-25T02:36:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 12434, 2023-08-25T02:36:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 2626225, 2023-08-25T02:36:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 13992558, 2023-08-25T02:36:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 51, 2023-08-25T02:36:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 6, 2023-08-25T02:36:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.536052137, 2023-08-25T02:36:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.210587827, 2023-08-25T02:36:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 1.176567971, 2023-08-25T02:36:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.084167630, 2023-08-25T02:36:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 12432, 2023-08-25T02:36:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2.209025624, 2023-08-25T02:36:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000177688 2023-08-25T02:36:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 16158, 2023-08-25T02:36:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.003112897, 2023-08-25T02:36:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000192 2023-08-25T02:36:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 45360, 2023-08-25T02:36:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 115520 2023-08-25T02:36:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 283364, 2023-08-25T02:36:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 10670, 2023-08-25T02:36:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 47280371, 2023-08-25T02:36:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 15775133, 2023-08-25T02:36:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 16, 2023-08-25T02:36:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 6, 2023-08-25T02:36:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 4.999512103, 2023-08-25T02:36:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 1.720150273, 2023-08-25T02:36:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 4.559280351, 2023-08-25T02:36:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.059660134, 2023-08-25T02:36:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 10668, 2023-08-25T02:36:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.925821357, 2023-08-25T02:36:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000086784 2023-08-25T02:36:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 290119, 2023-08-25T02:36:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.021844724, 2023-08-25T02:36:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000075 2023-08-25T02:36:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 160, 2023-08-25T02:36:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 305776 2023-08-25T02:36:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2023-08-25T02:36:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2023-08-25T02:36:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2023-08-25T02:36:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2023-08-25T02:36:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2717, 2023-08-25T02:36:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 4.151381440, 2023-08-25T02:36:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001527928 2023-08-25T02:36:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2023-08-25T02:36:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 95, 2023-08-25T02:36:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.003005850, 2023-08-25T02:36:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000031640 2023-08-25T02:36:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2023-08-25T02:36:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 3032, 2023-08-25T02:36:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 2633, 2023-08-25T02:36:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2023-08-25T02:36:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2633, 2023-08-25T02:36:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 70.425478848, 2023-08-25T02:36:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.026747238 2023-08-25T02:36:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2023-08-25T02:36:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 399, 2023-08-25T02:36:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 2, 2023-08-25T02:36:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2023-08-25T02:36:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 329, 2023-08-25T02:36:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 11, 2023-08-25T02:36:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2023-08-25T02:36:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 2428, 2023-08-25T02:36:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 1089, 2023-08-25T02:36:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 1324, 2023-08-25T02:36:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 15, 2023-08-25T02:36:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 702, 2023-08-25T02:36:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 1331, 2023-08-25T02:36:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 36, 2023-08-25T02:36:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 37, 2023-08-25T02:36:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 65, 2023-08-25T02:36:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 5606, 2023-08-25T02:36:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 4077, 2023-08-25T02:36:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 299, 2023-08-25T02:36:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 55, 2023-08-25T02:36:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2023-08-25T02:36:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 3, 2023-08-25T02:36:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 423, 2023-08-25T02:36:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2023-08-25T02:36:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2023-08-25T02:36:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 42, 2023-08-25T02:36:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 1184, 2023-08-25T02:36:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 44, 2023-08-25T02:36:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 1371, 2023-08-25T02:36:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2023-08-25T02:36:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:36:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 3, 2023-08-25T02:36:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 1044, 2023-08-25T02:36:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 7342262622, 2023-08-25T02:36:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 6, 2023-08-25T02:36:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:36:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2023-08-25T02:36:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:36:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:36:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:36:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:36:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:36:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2023-08-25T02:36:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 27, 2023-08-25T02:36:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 1535, 2023-08-25T02:36:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 3340, 2023-08-25T02:36:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:36:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 1, 2023-08-25T02:36:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:36:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 3314, 2023-08-25T02:36:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 1145, 2023-08-25T02:36:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 43, 2023-08-25T02:36:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 462 2023-08-25T02:36:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2023-08-25T02:36:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 90, 2023-08-25T02:36:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 0, 2023-08-25T02:36:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 0, 2023-08-25T02:36:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 61, 2023-08-25T02:36:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 204, 2023-08-25T02:36:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 0, 2023-08-25T02:36:49.364 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2023-08-25T02:36:49.365 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:36:49.365 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 80, 2023-08-25T02:36:49.365 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 27, 2023-08-25T02:36:49.366 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 189, 2023-08-25T02:36:49.366 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 12, 2023-08-25T02:36:49.367 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2023-08-25T02:36:49.367 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:36:49.368 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:36:49.368 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2023-08-25T02:36:49.368 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2023-08-25T02:36:49.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:36:49.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 49, 2023-08-25T02:36:49.370 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2023-08-25T02:36:49.370 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 11, 2023-08-25T02:36:49.371 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2023-08-25T02:36:49.371 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2023-08-25T02:36:49.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.372 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2023-08-25T02:36:49.372 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2023-08-25T02:36:49.372 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 8294, 2023-08-25T02:36:49.373 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 3473, 2023-08-25T02:36:49.373 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 3473, 2023-08-25T02:36:49.373 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 4821, 2023-08-25T02:36:49.373 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2023-08-25T02:36:49.374 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 6, 2023-08-25T02:36:49.374 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 35, 2023-08-25T02:36:49.374 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 27, 2023-08-25T02:36:49.374 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 27, 2023-08-25T02:36:49.374 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 8, 2023-08-25T02:36:49.375 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 1, 2023-08-25T02:36:49.375 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2023-08-25T02:36:49.375 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 1, 2023-08-25T02:36:49.375 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 14742073, 2023-08-25T02:36:49.376 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 31631013, 2023-08-25T02:36:49.376 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 0, 2023-08-25T02:36:49.376 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2023-08-25T02:36:49.377 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2372, 2023-08-25T02:36:49.377 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 24.052383065, 2023-08-25T02:36:49.378 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.010140127 2023-08-25T02:36:49.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.378 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 0 2023-08-25T02:36:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2023-08-25T02:36:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1958, 2023-08-25T02:36:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 3103, 2023-08-25T02:36:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 1145, 2023-08-25T02:36:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 185, 2023-08-25T02:36:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 218, 2023-08-25T02:36:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 33, 2023-08-25T02:36:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 2428, 2023-08-25T02:36:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 4151, 2023-08-25T02:36:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 1723, 2023-08-25T02:36:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 37, 2023-08-25T02:36:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 1954, 2023-08-25T02:36:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 1917, 2023-08-25T02:36:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 84252, 2023-08-25T02:36:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 207152 2023-08-25T02:36:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2023-08-25T02:36:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 4618, 2023-08-25T02:36:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 3426, 2023-08-25T02:36:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 3032, 2023-08-25T02:36:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 371, 2023-08-25T02:36:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 6197, 2023-08-25T02:36:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2023-08-25T02:36:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1037, 2023-08-25T02:36:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 49.832047506, 2023-08-25T02:36:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.048054047 2023-08-25T02:36:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2023-08-25T02:36:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 373, 2023-08-25T02:36:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 7.618719655, 2023-08-25T02:36:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.020425521 2023-08-25T02:36:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2023-08-25T02:36:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2023-08-25T02:36:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 55, 2023-08-25T02:36:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.195858545, 2023-08-25T02:36:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.003561064 2023-08-25T02:36:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2023-08-25T02:36:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 147, 2023-08-25T02:36:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.365746775, 2023-08-25T02:36:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.002488073 2023-08-25T02:36:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2023-08-25T02:36:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2023-08-25T02:36:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2023-08-25T02:36:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2023-08-25T02:36:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2023-08-25T02:36:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2023-08-25T02:36:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 10, 2023-08-25T02:36:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.001555013, 2023-08-25T02:36:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000155501 2023-08-25T02:36:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2023-08-25T02:36:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 28, 2023-08-25T02:36:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.631362502, 2023-08-25T02:36:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.022548660 2023-08-25T02:36:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2023-08-25T02:36:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2023-08-25T02:36:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 10, 2023-08-25T02:36:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 3.992175295, 2023-08-25T02:36:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.399217529 2023-08-25T02:36:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2023-08-25T02:36:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 541, 2023-08-25T02:36:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1.996379868, 2023-08-25T02:36:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.003690166 2023-08-25T02:36:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2023-08-25T02:36:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 142, 2023-08-25T02:36:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2.071584017, 2023-08-25T02:36:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.014588619 2023-08-25T02:36:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2023-08-25T02:36:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 211, 2023-08-25T02:36:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2.302724699, 2023-08-25T02:36:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.010913387 2023-08-25T02:36:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2023-08-25T02:36:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2023-08-25T02:36:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2023-08-25T02:36:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2023-08-25T02:36:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4, 2023-08-25T02:36:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.617060986, 2023-08-25T02:36:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.154265246 2023-08-25T02:36:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2023-08-25T02:36:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2023-08-25T02:36:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1, 2023-08-25T02:36:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.001243739, 2023-08-25T02:36:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001243739 2023-08-25T02:36:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2023-08-25T02:36:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2023-08-25T02:36:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2023-08-25T02:36:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2023-08-25T02:36:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2023-08-25T02:36:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2023-08-25T02:36:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 74, 2023-08-25T02:36:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.799020248, 2023-08-25T02:36:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.010797570 2023-08-25T02:36:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:36:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:36:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2023-08-25T02:36:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2023-08-25T02:36:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2023-08-25T02:36:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 3, 2023-08-25T02:36:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 1, 2023-08-25T02:36:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 2, 2023-08-25T02:36:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2023-08-25T02:36:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 43, 2023-08-25T02:36:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 21, 2023-08-25T02:36:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 603 2023-08-25T02:36:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2023-08-25T02:36:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2023-08-25T02:36:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 663594, 2023-08-25T02:36:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 1485, 2023-08-25T02:36:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2023-08-25T02:36:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2023-08-25T02:36:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8224, 2023-08-25T02:36:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 171, 2023-08-25T02:36:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2023-08-25T02:36:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 6519718, 2023-08-25T02:36:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 77435, 2023-08-25T02:36:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2023-08-25T02:36:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2023-08-25T02:36:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2023-08-25T02:36:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2023-08-25T02:36:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2023-08-25T02:36:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 4062, 2023-08-25T02:36:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 30602456, 2023-08-25T02:36:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 37, 2023-08-25T02:36:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 4062, 2023-08-25T02:36:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2023-08-25T02:36:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4062, 2023-08-25T02:36:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 84.716890623, 2023-08-25T02:36:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.020855955 2023-08-25T02:36:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2023-08-25T02:36:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2023-08-25T02:36:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4062, 2023-08-25T02:36:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 6822 2023-08-25T02:36:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 4062, 2023-08-25T02:36:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 12, 2023-08-25T02:36:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 4050, 2023-08-25T02:36:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2023-08-25T02:36:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2023-08-25T02:36:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 201, 2023-08-25T02:36:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 1081, 2023-08-25T02:36:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 10, 2023-08-25T02:36:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 2382, 2023-08-25T02:36:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 114, 2023-08-25T02:36:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2023-08-25T02:36:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2023-08-25T02:36:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 0, 2023-08-25T02:36:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 3, 2023-08-25T02:36:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 55, 2023-08-25T02:36:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2023-08-25T02:36:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2023-08-25T02:36:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2023-08-25T02:36:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 1, 2023-08-25T02:36:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 2162, 2023-08-25T02:36:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:36:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2023-08-25T02:36:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:36:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2023-08-25T02:36:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2023-08-25T02:36:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2023-08-25T02:36:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:36:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2023-08-25T02:36:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:36:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 0, 2023-08-25T02:36:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2023-08-25T02:36:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2023-08-25T02:36:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2023-08-25T02:36:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2023-08-25T02:36:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2023-08-25T02:36:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2023-08-25T02:36:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2023-08-25T02:36:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2023-08-25T02:36:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2023-08-25T02:36:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2023-08-25T02:36:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2023-08-25T02:36:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2023-08-25T02:36:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2023-08-25T02:36:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2023-08-25T02:36:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2023-08-25T02:36:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2023-08-25T02:36:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 91, 2023-08-25T02:36:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2023-08-25T02:36:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 12, 2023-08-25T02:36:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 12, 2023-08-25T02:36:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 12, 2023-08-25T02:36:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2023-08-25T02:36:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2023-08-25T02:36:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 741, 2023-08-25T02:36:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 4, 2023-08-25T02:36:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 68 2023-08-25T02:36:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2023-08-25T02:36:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 1, 2023-08-25T02:36:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:36:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2023-08-25T02:36:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2023-08-25T02:36:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2023-08-25T02:36:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 2161 2023-08-25T02:36:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2023-08-25T02:36:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2023-08-25T02:36:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 9, 2023-08-25T02:36:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2023-08-25T02:36:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 9, 2023-08-25T02:36:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 27, 2023-08-25T02:36:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2023-08-25T02:36:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 297760, 2023-08-25T02:36:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 27231999, 2023-08-25T02:36:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 297760, 2023-08-25T02:36:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 297760, 2023-08-25T02:36:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 27231999, 2023-08-25T02:36:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2023-08-25T02:36:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 4062, 2023-08-25T02:36:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 30605247, 2023-08-25T02:36:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 4062, 2023-08-25T02:36:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 30605247, 2023-08-25T02:36:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2023-08-25T02:36:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2023-08-25T02:36:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 4062, 2023-08-25T02:36:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 4062, 2023-08-25T02:36:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 4062, 2023-08-25T02:36:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 4062, 2023-08-25T02:36:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:36:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 27, 2023-08-25T02:36:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 2823, 2023-08-25T02:36:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 27, 2023-08-25T02:36:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 10, 2023-08-25T02:36:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 2823, 2023-08-25T02:36:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x55d677f2e6e0": { 2023-08-25T02:36:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 8294, 2023-08-25T02:36:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 27436709, 2023-08-25T02:36:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 8294, 2023-08-25T02:36:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2372, 2023-08-25T02:36:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 27436709, 2023-08-25T02:36:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:49.487 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-08-25T02:36:49.723 INFO:tasks.check_counter:Daemon mds.b mds.exported=42 2023-08-25T02:36:49.723 INFO:tasks.check_counter:Daemon mds.b mds.imported=44 2023-08-25T02:36:49.724 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=90 2023-08-25T02:36:49.724 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2023-08-25T02:36:49.724 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-08-25T02:36:49.725 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2023-08-25T02:36:49.725 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2023-08-25T02:36:49.726 DEBUG:tasks.check_counter:Getting stats from e 2023-08-25T02:36:49.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.e perf dump 2023-08-25T02:36:49.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:49 smithi121 ceph-mon[100099]: pgmap v320: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:49.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:49 smithi122 ceph-mon[100457]: pgmap v320: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:50.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:49 smithi089 ceph-mon[98732]: pgmap v320: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:50.277 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:36:51.492 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-08-25T02:36:51.493 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:51.493 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 300, 2023-08-25T02:36:51.493 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 213, 2023-08-25T02:36:51.494 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1485410, 2023-08-25T02:36:51.494 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 120365, 2023-08-25T02:36:51.494 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:51.495 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:51.495 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.043122303, 2023-08-25T02:36:51.495 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.012678860, 2023-08-25T02:36:51.495 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.031399746, 2023-08-25T02:36:51.496 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.005579188, 2023-08-25T02:36:51.496 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:51.496 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 212, 2023-08-25T02:36:51.497 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.011579203, 2023-08-25T02:36:51.497 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000054618 2023-08-25T02:36:51.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.498 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:51.498 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.498 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.499 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.499 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1484576, 2023-08-25T02:36:51.500 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 120048 2023-08-25T02:36:51.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.501 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:51.501 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 655, 2023-08-25T02:36:51.501 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 801, 2023-08-25T02:36:51.501 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 2858787, 2023-08-25T02:36:51.502 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 616868, 2023-08-25T02:36:51.502 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 7, 2023-08-25T02:36:51.502 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2023-08-25T02:36:51.503 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.064453774, 2023-08-25T02:36:51.503 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.021695591, 2023-08-25T02:36:51.503 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.015691051, 2023-08-25T02:36:51.504 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.019518763, 2023-08-25T02:36:51.504 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:51.505 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 799, 2023-08-25T02:36:51.505 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.034830931, 2023-08-25T02:36:51.505 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000043593 2023-08-25T02:36:51.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.506 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:51.506 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.507 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.507 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.508 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-08-25T02:36:51.508 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 415024 2023-08-25T02:36:51.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.509 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:51.509 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 1586, 2023-08-25T02:36:51.509 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 1588, 2023-08-25T02:36:51.510 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 4654689, 2023-08-25T02:36:51.510 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 493304, 2023-08-25T02:36:51.510 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:51.510 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 4, 2023-08-25T02:36:51.510 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.157510986, 2023-08-25T02:36:51.511 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.050528044, 2023-08-25T02:36:51.511 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.038931030, 2023-08-25T02:36:51.511 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.053995253, 2023-08-25T02:36:51.512 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:51.512 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1586, 2023-08-25T02:36:51.512 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.059136059, 2023-08-25T02:36:51.513 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000037286 2023-08-25T02:36:51.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.514 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:51.514 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.514 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.514 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.515 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:36:51.515 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:36:51.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.516 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2023-08-25T02:36:51.516 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2023-08-25T02:36:51.517 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2023-08-25T02:36:51.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.518 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2023-08-25T02:36:51.518 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:51.518 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:51.519 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3820, 2023-08-25T02:36:51.519 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.112175821, 2023-08-25T02:36:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000029365 2023-08-25T02:36:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2023-08-25T02:36:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2023-08-25T02:36:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 0, 2023-08-25T02:36:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 0, 2023-08-25T02:36:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2023-08-25T02:36:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2023-08-25T02:36:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 0, 2023-08-25T02:36:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 0, 2023-08-25T02:36:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2023-08-25T02:36:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 0, 2023-08-25T02:36:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 0, 2023-08-25T02:36:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2023-08-25T02:36:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 878, 2023-08-25T02:36:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 0, 2023-08-25T02:36:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 0, 2023-08-25T02:36:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 878, 2023-08-25T02:36:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 878, 2023-08-25T02:36:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 0, 2023-08-25T02:36:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 0, 2023-08-25T02:36:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 0, 2023-08-25T02:36:51.532 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 49, 2023-08-25T02:36:51.532 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 0, 2023-08-25T02:36:51.532 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 0, 2023-08-25T02:36:51.532 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 0, 2023-08-25T02:36:51.532 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 0, 2023-08-25T02:36:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2023-08-25T02:36:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 0, 2023-08-25T02:36:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 0, 2023-08-25T02:36:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2023-08-25T02:36:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2023-08-25T02:36:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 0, 2023-08-25T02:36:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 0, 2023-08-25T02:36:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 0, 2023-08-25T02:36:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 0, 2023-08-25T02:36:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2023-08-25T02:36:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:36:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 0, 2023-08-25T02:36:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 0, 2023-08-25T02:36:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 0, 2023-08-25T02:36:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 0, 2023-08-25T02:36:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:36:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2023-08-25T02:36:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:36:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:36:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:36:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:36:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:36:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2023-08-25T02:36:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 0, 2023-08-25T02:36:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 0, 2023-08-25T02:36:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 0, 2023-08-25T02:36:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:36:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-08-25T02:36:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:36:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 0, 2023-08-25T02:36:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 0, 2023-08-25T02:36:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 0, 2023-08-25T02:36:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 0 2023-08-25T02:36:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2023-08-25T02:36:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 0, 2023-08-25T02:36:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 0, 2023-08-25T02:36:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 0, 2023-08-25T02:36:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 0, 2023-08-25T02:36:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 0, 2023-08-25T02:36:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 0, 2023-08-25T02:36:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2023-08-25T02:36:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:36:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 0, 2023-08-25T02:36:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 0, 2023-08-25T02:36:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 0, 2023-08-25T02:36:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 0, 2023-08-25T02:36:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2023-08-25T02:36:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:36:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2023-08-25T02:36:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 0, 2023-08-25T02:36:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2023-08-25T02:36:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2023-08-25T02:36:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2023-08-25T02:36:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2023-08-25T02:36:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 0, 2023-08-25T02:36:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 0, 2023-08-25T02:36:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 0, 2023-08-25T02:36:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 4474, 2023-08-25T02:36:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2023-08-25T02:36:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 0, 2023-08-25T02:36:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 0, 2023-08-25T02:36:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 0, 2023-08-25T02:36:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 0, 2023-08-25T02:36:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 6, 2023-08-25T02:36:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 2, 2023-08-25T02:36:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2023-08-25T02:36:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 0, 2023-08-25T02:36:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 4194304, 2023-08-25T02:36:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 11080167, 2023-08-25T02:36:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 11079593, 2023-08-25T02:36:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2023-08-25T02:36:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 4474 2023-08-25T02:36:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2023-08-25T02:36:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 796, 2023-08-25T02:36:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 1415, 2023-08-25T02:36:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 619, 2023-08-25T02:36:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 56, 2023-08-25T02:36:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 97, 2023-08-25T02:36:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 41, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 878, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 1735, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 857, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 0, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 0, 2023-08-25T02:36:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 0, 2023-08-25T02:36:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 52388, 2023-08-25T02:36:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 207152 2023-08-25T02:36:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2023-08-25T02:36:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 0, 2023-08-25T02:36:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 0, 2023-08-25T02:36:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 0, 2023-08-25T02:36:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 0, 2023-08-25T02:36:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 0, 2023-08-25T02:36:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2023-08-25T02:36:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2023-08-25T02:36:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2023-08-25T02:36:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2023-08-25T02:36:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2023-08-25T02:36:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2023-08-25T02:36:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2023-08-25T02:36:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2023-08-25T02:36:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2023-08-25T02:36:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2023-08-25T02:36:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2023-08-25T02:36:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2023-08-25T02:36:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2023-08-25T02:36:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2023-08-25T02:36:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2023-08-25T02:36:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2023-08-25T02:36:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2023-08-25T02:36:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2023-08-25T02:36:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2023-08-25T02:36:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2023-08-25T02:36:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2023-08-25T02:36:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2023-08-25T02:36:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2023-08-25T02:36:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2023-08-25T02:36:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2023-08-25T02:36:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2023-08-25T02:36:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2023-08-25T02:36:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:36:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:36:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2023-08-25T02:36:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2023-08-25T02:36:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2023-08-25T02:36:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 4, 2023-08-25T02:36:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 2, 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 4, 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 0, 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 0, 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 3684 2023-08-25T02:36:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2023-08-25T02:36:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2023-08-25T02:36:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 228577, 2023-08-25T02:36:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 877, 2023-08-25T02:36:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8224, 2023-08-25T02:36:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 171, 2023-08-25T02:36:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2023-08-25T02:36:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 2518332, 2023-08-25T02:36:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 25011, 2023-08-25T02:36:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2023-08-25T02:36:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2023-08-25T02:36:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2023-08-25T02:36:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2023-08-25T02:36:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2023-08-25T02:36:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 2236, 2023-08-25T02:36:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 21, 2023-08-25T02:36:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 3, 2023-08-25T02:36:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 2236, 2023-08-25T02:36:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2023-08-25T02:36:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2236, 2023-08-25T02:36:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1.196984273, 2023-08-25T02:36:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000535323 2023-08-25T02:36:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2023-08-25T02:36:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2023-08-25T02:36:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2236, 2023-08-25T02:36:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2237 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 2236, 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 2236, 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 0, 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2023-08-25T02:36:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2023-08-25T02:36:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 1321, 2023-08-25T02:36:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 0, 2023-08-25T02:36:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 913, 2023-08-25T02:36:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 0, 2023-08-25T02:36:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 0, 2023-08-25T02:36:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2023-08-25T02:36:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2023-08-25T02:36:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 0, 2023-08-25T02:36:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 0, 2023-08-25T02:36:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 0, 2023-08-25T02:36:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2023-08-25T02:36:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2023-08-25T02:36:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2023-08-25T02:36:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 1, 2023-08-25T02:36:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 0, 2023-08-25T02:36:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:36:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2023-08-25T02:36:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:36:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2023-08-25T02:36:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 0, 2023-08-25T02:36:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2023-08-25T02:36:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2023-08-25T02:36:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2023-08-25T02:36:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2023-08-25T02:36:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2023-08-25T02:36:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2023-08-25T02:36:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2023-08-25T02:36:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2023-08-25T02:36:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2023-08-25T02:36:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2023-08-25T02:36:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2023-08-25T02:36:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2023-08-25T02:36:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2023-08-25T02:36:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2023-08-25T02:36:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2023-08-25T02:36:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2023-08-25T02:36:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 91, 2023-08-25T02:36:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2023-08-25T02:36:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 11, 2023-08-25T02:36:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 6, 2023-08-25T02:36:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 6, 2023-08-25T02:36:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2023-08-25T02:36:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2023-08-25T02:36:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 0, 2023-08-25T02:36:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 2, 2023-08-25T02:36:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 0 2023-08-25T02:36:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2023-08-25T02:36:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 0, 2023-08-25T02:36:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:36:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2023-08-25T02:36:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2023-08-25T02:36:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2023-08-25T02:36:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 0 2023-08-25T02:36:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2023-08-25T02:36:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2023-08-25T02:36:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 0, 2023-08-25T02:36:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2023-08-25T02:36:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 0, 2023-08-25T02:36:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 0, 2023-08-25T02:36:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2023-08-25T02:36:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 2537, 2023-08-25T02:36:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 8752818, 2023-08-25T02:36:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 2537, 2023-08-25T02:36:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2537, 2023-08-25T02:36:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 8752818, 2023-08-25T02:36:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2023-08-25T02:36:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 2236, 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 6885868, 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2236, 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 6885868, 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2023-08-25T02:36:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2023-08-25T02:36:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 2236, 2023-08-25T02:36:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 2236, 2023-08-25T02:36:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2236, 2023-08-25T02:36:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 2236, 2023-08-25T02:36:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:36:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 0, 2023-08-25T02:36:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 0, 2023-08-25T02:36:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x55f5b594e960": { 2023-08-25T02:36:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 0, 2023-08-25T02:36:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 0, 2023-08-25T02:36:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:51.633 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-08-25T02:36:51.634 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:51 smithi089 ceph-mon[98732]: pgmap v321: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:51.708 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:51 smithi121 ceph-mon[100099]: pgmap v321: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:51.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:51 smithi122 ceph-mon[100457]: pgmap v321: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:51.929 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-08-25T02:36:51.930 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-08-25T02:36:51.930 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-08-25T02:36:51.930 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-08-25T02:36:51.931 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2023-08-25T02:36:51.931 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-08-25T02:36:51.931 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-08-25T02:36:51.932 DEBUG:tasks.check_counter:Getting stats from h 2023-08-25T02:36:51.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.h perf dump 2023-08-25T02:36:52.483 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:36:52.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:52 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:36:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:52 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:36:52.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:52 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-08-25T02:36:53.722 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-08-25T02:36:53.723 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:53.723 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 3886, 2023-08-25T02:36:53.723 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 5840, 2023-08-25T02:36:53.723 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1114375, 2023-08-25T02:36:53.723 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 7213699, 2023-08-25T02:36:53.724 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 15, 2023-08-25T02:36:53.724 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2023-08-25T02:36:53.724 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.249197408, 2023-08-25T02:36:53.724 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.104291393, 2023-08-25T02:36:53.724 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.222558937, 2023-08-25T02:36:53.725 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.027362297, 2023-08-25T02:36:53.725 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:53.725 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 5837, 2023-08-25T02:36:53.725 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.328117645, 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000056213 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 6126, 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.001424249, 2023-08-25T02:36:53.726 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000232 2023-08-25T02:36:53.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.727 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3904, 2023-08-25T02:36:53.727 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 1376 2023-08-25T02:36:53.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.727 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:53.728 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 17442, 2023-08-25T02:36:53.728 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 16317, 2023-08-25T02:36:53.728 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 5726739, 2023-08-25T02:36:53.728 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 5961935, 2023-08-25T02:36:53.728 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 7, 2023-08-25T02:36:53.729 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 6, 2023-08-25T02:36:53.729 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.735706643, 2023-08-25T02:36:53.729 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.284784136, 2023-08-25T02:36:53.729 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.564121854, 2023-08-25T02:36:53.729 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.013855890, 2023-08-25T02:36:53.730 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:53.730 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 16316, 2023-08-25T02:36:53.730 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.307188848, 2023-08-25T02:36:53.730 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000018827 2023-08-25T02:36:53.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.731 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:53.731 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 31954, 2023-08-25T02:36:53.731 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.004838519, 2023-08-25T02:36:53.731 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000151 2023-08-25T02:36:53.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.732 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1494864, 2023-08-25T02:36:53.732 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 130368 2023-08-25T02:36:53.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.732 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:53.732 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 3783, 2023-08-25T02:36:53.733 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 3895, 2023-08-25T02:36:53.733 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 2154785, 2023-08-25T02:36:53.733 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 2663260, 2023-08-25T02:36:53.733 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 11, 2023-08-25T02:36:53.733 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2023-08-25T02:36:53.734 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.267235824, 2023-08-25T02:36:53.734 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.130081153, 2023-08-25T02:36:53.734 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.158777270, 2023-08-25T02:36:53.734 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.004778727, 2023-08-25T02:36:53.735 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:53.735 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3894, 2023-08-25T02:36:53.735 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.122352781, 2023-08-25T02:36:53.735 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000031420 2023-08-25T02:36:53.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.736 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:53.736 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 7170, 2023-08-25T02:36:53.736 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.001510657, 2023-08-25T02:36:53.736 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000210 2023-08-25T02:36:53.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1936, 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 418096 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2023-08-25T02:36:53.737 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2023-08-25T02:36:53.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.738 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2023-08-25T02:36:53.738 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:53.738 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:53.739 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1878, 2023-08-25T02:36:53.739 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1.184135727, 2023-08-25T02:36:53.739 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000630530 2023-08-25T02:36:53.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.740 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2023-08-25T02:36:53.740 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2023-08-25T02:36:53.740 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2023-08-25T02:36:53.740 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 98, 2023-08-25T02:36:53.740 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.004059906, 2023-08-25T02:36:53.741 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000041427 2023-08-25T02:36:53.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.741 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2023-08-25T02:36:53.741 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 1198, 2023-08-25T02:36:53.742 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 1003, 2023-08-25T02:36:53.742 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2023-08-25T02:36:53.742 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1003, 2023-08-25T02:36:53.742 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 734.405223308, 2023-08-25T02:36:53.742 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.732208597 2023-08-25T02:36:53.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.743 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2023-08-25T02:36:53.743 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 195, 2023-08-25T02:36:53.743 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 1, 2023-08-25T02:36:53.744 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2023-08-25T02:36:53.744 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 67, 2023-08-25T02:36:53.744 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 7, 2023-08-25T02:36:53.744 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2023-08-25T02:36:53.745 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 568, 2023-08-25T02:36:53.745 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 217, 2023-08-25T02:36:53.745 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 245, 2023-08-25T02:36:53.745 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 106, 2023-08-25T02:36:53.745 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 225, 2023-08-25T02:36:53.746 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 1431, 2023-08-25T02:36:53.746 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 7, 2023-08-25T02:36:53.746 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 7, 2023-08-25T02:36:53.746 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 61, 2023-08-25T02:36:53.746 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 18608, 2023-08-25T02:36:53.747 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 13805, 2023-08-25T02:36:53.747 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 186, 2023-08-25T02:36:53.747 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 94, 2023-08-25T02:36:53.747 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2023-08-25T02:36:53.747 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 2, 2023-08-25T02:36:53.748 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 3811, 2023-08-25T02:36:53.748 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2023-08-25T02:36:53.748 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2023-08-25T02:36:53.748 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 28, 2023-08-25T02:36:53.749 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 983, 2023-08-25T02:36:53.749 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 26, 2023-08-25T02:36:53.749 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 963, 2023-08-25T02:36:53.749 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2023-08-25T02:36:53.749 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:36:53.750 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 3, 2023-08-25T02:36:53.750 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 1044, 2023-08-25T02:36:53.750 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 7342262210, 2023-08-25T02:36:53.750 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 6, 2023-08-25T02:36:53.751 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:36:53.751 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2023-08-25T02:36:53.751 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:36:53.751 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:36:53.751 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:36:53.752 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:36:53.752 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:36:53.752 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2023-08-25T02:36:53.752 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 61, 2023-08-25T02:36:53.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 911, 2023-08-25T02:36:53.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 1560, 2023-08-25T02:36:53.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:36:53.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-08-25T02:36:53.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:36:53.754 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 1588, 2023-08-25T02:36:53.754 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 285, 2023-08-25T02:36:53.754 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 54, 2023-08-25T02:36:53.754 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 336 2023-08-25T02:36:53.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.755 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2023-08-25T02:36:53.755 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 25, 2023-08-25T02:36:53.755 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 133, 2023-08-25T02:36:53.756 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 75, 2023-08-25T02:36:53.756 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 176, 2023-08-25T02:36:53.756 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 149, 2023-08-25T02:36:53.756 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 4, 2023-08-25T02:36:53.756 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2023-08-25T02:36:53.757 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:36:53.757 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 67, 2023-08-25T02:36:53.757 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 12, 2023-08-25T02:36:53.757 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 229, 2023-08-25T02:36:53.757 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 38, 2023-08-25T02:36:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2023-08-25T02:36:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:36:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:36:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2023-08-25T02:36:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2023-08-25T02:36:53.759 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:36:53.759 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 28, 2023-08-25T02:36:53.759 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2023-08-25T02:36:53.759 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 7, 2023-08-25T02:36:53.759 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2023-08-25T02:36:53.760 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2023-08-25T02:36:53.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.760 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2023-08-25T02:36:53.760 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2023-08-25T02:36:53.760 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 4474, 2023-08-25T02:36:53.761 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 0, 2023-08-25T02:36:53.761 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 0, 2023-08-25T02:36:53.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 4474, 2023-08-25T02:36:53.761 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2023-08-25T02:36:53.761 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 0, 2023-08-25T02:36:53.762 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 6, 2023-08-25T02:36:53.762 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 0, 2023-08-25T02:36:53.762 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 0, 2023-08-25T02:36:53.762 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 6, 2023-08-25T02:36:53.762 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 2, 2023-08-25T02:36:53.763 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2023-08-25T02:36:53.763 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 0, 2023-08-25T02:36:53.763 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 4194304, 2023-08-25T02:36:53.763 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 11080167, 2023-08-25T02:36:53.763 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 0, 2023-08-25T02:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2023-08-25T02:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1114, 2023-08-25T02:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 9.063699594, 2023-08-25T02:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.008136175 2023-08-25T02:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.765 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 0 2023-08-25T02:36:53.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.765 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2023-08-25T02:36:53.765 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 493, 2023-08-25T02:36:53.765 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 1430, 2023-08-25T02:36:53.766 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 937, 2023-08-25T02:36:53.766 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 164, 2023-08-25T02:36:53.766 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 229, 2023-08-25T02:36:53.766 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 65, 2023-08-25T02:36:53.766 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 569, 2023-08-25T02:36:53.767 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 2049, 2023-08-25T02:36:53.767 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 1480, 2023-08-25T02:36:53.767 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 7, 2023-08-25T02:36:53.767 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 996, 2023-08-25T02:36:53.767 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 989, 2023-08-25T02:36:53.768 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 65592, 2023-08-25T02:36:53.768 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 198960 2023-08-25T02:36:53.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.768 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2023-08-25T02:36:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 10397, 2023-08-25T02:36:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 1804, 2023-08-25T02:36:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 1198, 2023-08-25T02:36:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 131, 2023-08-25T02:36:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 5166, 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 32, 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 10.851674739, 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.339114835 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.770 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2023-08-25T02:36:53.771 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 15, 2023-08-25T02:36:53.771 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.006622141, 2023-08-25T02:36:53.771 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000441476 2023-08-25T02:36:53.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.771 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2023-08-25T02:36:53.772 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.772 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.772 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.772 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2023-08-25T02:36:53.773 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 46, 2023-08-25T02:36:53.773 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2.704812457, 2023-08-25T02:36:53.773 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.058800270 2023-08-25T02:36:53.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.773 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2023-08-25T02:36:53.774 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 385, 2023-08-25T02:36:53.774 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 6.598227419, 2023-08-25T02:36:53.774 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.017138253 2023-08-25T02:36:53.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.774 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2023-08-25T02:36:53.775 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.775 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.775 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.775 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2023-08-25T02:36:53.776 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.776 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.776 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.776 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2023-08-25T02:36:53.777 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.778 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.778 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.778 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2023-08-25T02:36:53.778 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.779 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.779 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.779 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2023-08-25T02:36:53.779 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.780 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.780 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.780 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2023-08-25T02:36:53.780 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.781 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.781 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.781 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2023-08-25T02:36:53.781 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.782 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.782 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.782 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2023-08-25T02:36:53.783 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.783 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.783 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.783 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2023-08-25T02:36:53.784 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 47, 2023-08-25T02:36:53.784 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 8.507477704, 2023-08-25T02:36:53.784 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.181010163 2023-08-25T02:36:53.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.784 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2023-08-25T02:36:53.785 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 154, 2023-08-25T02:36:53.785 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 5.569663146, 2023-08-25T02:36:53.785 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.036166643 2023-08-25T02:36:53.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.785 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2023-08-25T02:36:53.786 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 261, 2023-08-25T02:36:53.786 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 657.642538529, 2023-08-25T02:36:53.786 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 2.519703212 2023-08-25T02:36:53.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.786 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2023-08-25T02:36:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2023-08-25T02:36:53.788 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.788 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.788 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.789 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2023-08-25T02:36:53.789 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.789 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.789 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.790 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2023-08-25T02:36:53.790 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.790 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.790 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.791 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2023-08-25T02:36:53.791 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.791 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.791 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.792 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2023-08-25T02:36:53.792 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.792 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.792 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.793 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2023-08-25T02:36:53.793 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.793 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.793 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.794 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2023-08-25T02:36:53.794 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.794 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.794 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.795 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2023-08-25T02:36:53.795 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.795 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.795 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.796 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2023-08-25T02:36:53.796 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.796 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.796 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.797 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2023-08-25T02:36:53.797 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.797 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.797 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.798 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2023-08-25T02:36:53.798 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 63, 2023-08-25T02:36:53.798 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 42.524207173, 2023-08-25T02:36:53.799 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.674987415 2023-08-25T02:36:53.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.799 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:36:53.799 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:36:53.799 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2023-08-25T02:36:53.800 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.800 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.800 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.801 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2023-08-25T02:36:53.801 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2023-08-25T02:36:53.801 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 4, 2023-08-25T02:36:53.801 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 2, 2023-08-25T02:36:53.801 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 2, 2023-08-25T02:36:53.802 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2023-08-25T02:36:53.802 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 189, 2023-08-25T02:36:53.802 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 94, 2023-08-25T02:36:53.802 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 601 2023-08-25T02:36:53.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.803 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2023-08-25T02:36:53.803 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:53.803 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:53.803 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:53.803 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:53.804 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:53.804 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:53.804 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:53.804 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:53.805 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:53.805 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:53.805 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:53.805 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:53.805 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:53.806 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:53.806 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:53.806 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:53.806 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:53.806 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:53.807 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:53.808 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:53.808 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:53.808 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:53.808 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:53.808 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2023-08-25T02:36:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:53.809 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:53.810 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:53.810 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:53.810 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 508547, 2023-08-25T02:36:53.810 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 496, 2023-08-25T02:36:53.810 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2023-08-25T02:36:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2023-08-25T02:36:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:53.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:53.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:53.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8224, 2023-08-25T02:36:53.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 171, 2023-08-25T02:36:53.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:53.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:53.813 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:53.813 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2023-08-25T02:36:53.813 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 1810572, 2023-08-25T02:36:53.813 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 21573, 2023-08-25T02:36:53.814 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:53.814 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2023-08-25T02:36:53.814 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:53.814 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2023-08-25T02:36:53.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.815 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2023-08-25T02:36:53.815 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2023-08-25T02:36:53.815 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2023-08-25T02:36:53.815 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 1369, 2023-08-25T02:36:53.815 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 7499003, 2023-08-25T02:36:53.816 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 27, 2023-08-25T02:36:53.816 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 1369, 2023-08-25T02:36:53.816 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2023-08-25T02:36:53.816 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1369, 2023-08-25T02:36:53.816 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 8.758884738, 2023-08-25T02:36:53.817 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.006398016 2023-08-25T02:36:53.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.817 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2023-08-25T02:36:53.817 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2023-08-25T02:36:53.817 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1369, 2023-08-25T02:36:53.818 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1556 2023-08-25T02:36:53.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.818 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 1369, 2023-08-25T02:36:53.818 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 13, 2023-08-25T02:36:53.818 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 1356, 2023-08-25T02:36:53.819 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2023-08-25T02:36:53.819 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2023-08-25T02:36:53.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 0, 2023-08-25T02:36:53.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 10, 2023-08-25T02:36:53.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 12, 2023-08-25T02:36:53.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 1126, 2023-08-25T02:36:53.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 45, 2023-08-25T02:36:53.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2023-08-25T02:36:53.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2023-08-25T02:36:53.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 2, 2023-08-25T02:36:53.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 1, 2023-08-25T02:36:53.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 35, 2023-08-25T02:36:53.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2023-08-25T02:36:53.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2023-08-25T02:36:53.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2023-08-25T02:36:53.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 0, 2023-08-25T02:36:53.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 20, 2023-08-25T02:36:53.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:36:53.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2023-08-25T02:36:53.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:36:53.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2023-08-25T02:36:53.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2023-08-25T02:36:53.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2023-08-25T02:36:53.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:36:53.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2023-08-25T02:36:53.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:36:53.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 0, 2023-08-25T02:36:53.824 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2023-08-25T02:36:53.824 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2023-08-25T02:36:53.825 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2023-08-25T02:36:53.825 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2023-08-25T02:36:53.825 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2023-08-25T02:36:53.825 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2023-08-25T02:36:53.825 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2023-08-25T02:36:53.826 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2023-08-25T02:36:53.826 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2023-08-25T02:36:53.826 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2023-08-25T02:36:53.826 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2023-08-25T02:36:53.826 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2023-08-25T02:36:53.827 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2023-08-25T02:36:53.827 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2023-08-25T02:36:53.827 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2023-08-25T02:36:53.827 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2023-08-25T02:36:53.827 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 91, 2023-08-25T02:36:53.828 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2023-08-25T02:36:53.828 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 11, 2023-08-25T02:36:53.828 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 11, 2023-08-25T02:36:53.828 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 11, 2023-08-25T02:36:53.829 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2023-08-25T02:36:53.829 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2023-08-25T02:36:53.829 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 250, 2023-08-25T02:36:53.829 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 2, 2023-08-25T02:36:53.829 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 53 2023-08-25T02:36:53.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.830 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2023-08-25T02:36:53.830 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 1, 2023-08-25T02:36:53.830 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:36:53.831 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2023-08-25T02:36:53.831 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2023-08-25T02:36:53.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.831 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2023-08-25T02:36:53.831 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 32 2023-08-25T02:36:53.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.832 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2023-08-25T02:36:53.832 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2023-08-25T02:36:53.832 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 4, 2023-08-25T02:36:53.832 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2023-08-25T02:36:53.833 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 2, 2023-08-25T02:36:53.833 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 12, 2023-08-25T02:36:53.833 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2023-08-25T02:36:53.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.833 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:53.834 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:53.834 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:53.834 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:53.834 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 25107, 2023-08-25T02:36:53.834 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 6716560, 2023-08-25T02:36:53.835 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:53.835 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 25107, 2023-08-25T02:36:53.835 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:53.835 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:53.835 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 25107, 2023-08-25T02:36:53.836 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 6716560, 2023-08-25T02:36:53.836 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:53.836 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.836 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.837 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.837 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2023-08-25T02:36:53.837 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:53.838 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:53.838 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:53.838 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:53.838 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:53.838 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:53.839 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:53.839 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 1369, 2023-08-25T02:36:53.839 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 7500343, 2023-08-25T02:36:53.839 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 1369, 2023-08-25T02:36:53.839 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 7500343, 2023-08-25T02:36:53.840 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:53.840 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.840 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.840 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.841 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2023-08-25T02:36:53.841 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:53.841 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2023-08-25T02:36:53.841 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:53.842 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2023-08-25T02:36:53.842 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2023-08-25T02:36:53.842 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:53.842 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:53.843 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 1369, 2023-08-25T02:36:53.843 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 1369, 2023-08-25T02:36:53.843 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 1369, 2023-08-25T02:36:53.843 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1369, 2023-08-25T02:36:53.843 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:53.844 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.844 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.845 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.846 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:36:53.846 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:53.846 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:53.846 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:53.847 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 12, 2023-08-25T02:36:53.847 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1356, 2023-08-25T02:36:53.847 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:53.847 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 12, 2023-08-25T02:36:53.847 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:53.848 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:53.848 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 12, 2023-08-25T02:36:53.848 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1356, 2023-08-25T02:36:53.848 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:53.849 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.849 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.849 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.849 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:53.850 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x5601e087e6e0": { 2023-08-25T02:36:53.850 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:53.850 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2023-08-25T02:36:53.850 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:53.850 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 4474, 2023-08-25T02:36:53.851 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 6885863, 2023-08-25T02:36:53.851 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:53.851 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 4474, 2023-08-25T02:36:53.851 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:53.851 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:53.852 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 1114, 2023-08-25T02:36:53.852 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 6885863, 2023-08-25T02:36:53.852 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:53.852 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:53.852 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:53.853 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:53.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:53.853 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-08-25T02:36:53.854 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:53 smithi089 ceph-mon[98732]: pgmap v322: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:53.854 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:53 smithi121 ceph-mon[100099]: pgmap v322: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:53.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:53 smithi122 ceph-mon[100457]: pgmap v322: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:54.339 INFO:tasks.check_counter:Daemon mds.h mds.exported=28 2023-08-25T02:36:54.340 INFO:tasks.check_counter:Daemon mds.h mds.imported=26 2023-08-25T02:36:54.340 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=25 2023-08-25T02:36:54.340 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=133 2023-08-25T02:36:54.341 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2023-08-25T02:36:54.341 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2023-08-25T02:36:54.341 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2023-08-25T02:36:54.341 DEBUG:tasks.check_counter:Getting stats from k 2023-08-25T02:36:54.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.k perf dump 2023-08-25T02:36:54.900 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.b/config 2023-08-25T02:36:55.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:55 smithi121 ceph-mon[100099]: pgmap v323: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:55.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:55 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:55 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:55 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:55 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:55 smithi122 ceph-mon[100457]: pgmap v323: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:55 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:55 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:55 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:55.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:55 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:55 smithi089 ceph-mon[98732]: pgmap v323: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:56.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:55 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:55 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:55 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:55 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.187 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-08-25T02:36:56.188 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:56.188 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 0, 2023-08-25T02:36:56.188 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 2, 2023-08-25T02:36:56.188 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 0, 2023-08-25T02:36:56.189 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 0, 2023-08-25T02:36:56.189 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2023-08-25T02:36:56.189 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2023-08-25T02:36:56.189 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.001081803, 2023-08-25T02:36:56.189 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000000000, 2023-08-25T02:36:56.190 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000222549, 2023-08-25T02:36:56.190 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:56.190 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:56.190 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:56.191 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:56.191 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:56.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.191 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:56.192 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:56.192 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:56.192 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:56.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.192 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:36:56.193 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:36:56.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.193 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:56.193 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2023-08-25T02:36:56.194 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 152, 2023-08-25T02:36:56.194 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2023-08-25T02:36:56.194 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 179184, 2023-08-25T02:36:56.194 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:56.195 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2023-08-25T02:36:56.195 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.009914112, 2023-08-25T02:36:56.195 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.006004205, 2023-08-25T02:36:56.195 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001782906, 2023-08-25T02:36:56.195 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:36:56.196 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:56.196 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 149, 2023-08-25T02:36:56.196 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.018474472, 2023-08-25T02:36:56.196 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000123989 2023-08-25T02:36:56.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.197 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:56.197 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:56.198 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:56.198 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:56.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.199 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2023-08-25T02:36:56.199 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 179184 2023-08-25T02:36:56.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.199 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:56.200 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 211, 2023-08-25T02:36:56.200 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 201, 2023-08-25T02:36:56.200 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1141168, 2023-08-25T02:36:56.201 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 119600, 2023-08-25T02:36:56.201 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2023-08-25T02:36:56.201 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:56.201 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.038673447, 2023-08-25T02:36:56.201 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.012120759, 2023-08-25T02:36:56.202 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.026174900, 2023-08-25T02:36:56.202 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.005805778, 2023-08-25T02:36:56.202 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:56.203 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 201, 2023-08-25T02:36:56.203 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.010030545, 2023-08-25T02:36:56.203 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000049903 2023-08-25T02:36:56.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.204 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:56.204 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:56.204 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:56.204 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:56.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.205 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1141168, 2023-08-25T02:36:56.206 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 119600 2023-08-25T02:36:56.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.206 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2023-08-25T02:36:56.206 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2023-08-25T02:36:56.207 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2023-08-25T02:36:56.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.207 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2023-08-25T02:36:56.207 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:56.208 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:56.208 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:56.208 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:56.208 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:56.208 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:56.209 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:56.209 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:56.209 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:56.209 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:56.209 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:56.210 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:56.210 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:56.210 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:56.210 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:56.212 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:56.212 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:56.212 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:56.213 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:56.213 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:56.213 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:56.213 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:56.214 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:56.214 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:56.214 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:56.214 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:56.215 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:56.215 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2023-08-25T02:36:56.215 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:56.215 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:56.215 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:56.216 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:56.216 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 149017, 2023-08-25T02:36:56.216 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 31, 2023-08-25T02:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2023-08-25T02:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2023-08-25T02:36:56.217 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:56.218 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:56.218 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:56.218 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:56.218 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2023-08-25T02:36:56.219 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2023-08-25T02:36:56.219 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:56.219 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:56.220 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:56.220 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2023-08-25T02:36:56.220 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2023-08-25T02:36:56.220 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2023-08-25T02:36:56.221 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:56.221 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2023-08-25T02:36:56.221 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:56.221 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2023-08-25T02:36:56.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-08-25T02:36:56.222 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:56.222 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2023-08-25T02:36:56.222 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2023-08-25T02:36:56.222 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2023-08-25T02:36:56.223 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 209, 2023-08-25T02:36:56.223 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1108987, 2023-08-25T02:36:56.224 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:56.224 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 209, 2023-08-25T02:36:56.224 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2023-08-25T02:36:56.224 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2023-08-25T02:36:56.225 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 209, 2023-08-25T02:36:56.225 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1108987, 2023-08-25T02:36:56.225 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2023-08-25T02:36:56.225 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2023-08-25T02:36:56.225 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2023-08-25T02:36:56.226 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2023-08-25T02:36:56.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:56.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-08-25T02:36:56.226 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-08-25T02:36:56.656 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-08-25T02:36:56.657 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-08-25T02:36:56.657 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2023-08-25T02:36:56.657 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2023-08-25T02:36:56.658 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-08-25T02:36:56.658 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-08-25T02:36:56.658 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-08-25T02:36:56.658 DEBUG:tasks.check_counter:Getting stats from c 2023-08-25T02:36:56.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.c perf dump 2023-08-25T02:36:56.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:36:56.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:56.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:36:56.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:56.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:36:56.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:36:56.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:56 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:36:56.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:56.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:36:56.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:56.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:56.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:56.983 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:36:56.984 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:36:56.984 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:56 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:57.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi089 to 196.1M 2023-08-25T02:36:57.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi089 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:57.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.081 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-08-25T02:36:57.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-08-25T02:36:57.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:57.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:57.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-08-25T02:36:57.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-08-25T02:36:57.083 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-08-25T02:36:57.084 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:56 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' 2023-08-25T02:36:57.248 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:36:57.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:57 smithi122 ceph-mon[100457]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:36:57.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:57 smithi122 ceph-mon[100457]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:36:57.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:57 smithi122 ceph-mon[100457]: pgmap v324: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:57.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:57 smithi121 ceph-mon[100099]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:36:57.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:57 smithi121 ceph-mon[100099]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:36:57.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:57 smithi121 ceph-mon[100099]: pgmap v324: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:58.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:57 smithi089 ceph-mon[98732]: Adjusting osd_memory_target on smithi121 to 196.1M 2023-08-25T02:36:58.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:57 smithi089 ceph-mon[98732]: Unable to set osd_memory_target on smithi121 to 205707980: error parsing value: Value '205707980' is below minimum 939524096 2023-08-25T02:36:58.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:57 smithi089 ceph-mon[98732]: pgmap v324: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-08-25T02:36:58.549 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-08-25T02:36:58.550 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:36:58.550 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 1140, 2023-08-25T02:36:58.550 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 1051, 2023-08-25T02:36:58.550 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 10149193, 2023-08-25T02:36:58.551 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 380969, 2023-08-25T02:36:58.551 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:58.551 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 1, 2023-08-25T02:36:58.551 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.124354675, 2023-08-25T02:36:58.551 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.034702932, 2023-08-25T02:36:58.552 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.053211152, 2023-08-25T02:36:58.552 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.034784492, 2023-08-25T02:36:58.552 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:58.552 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1051, 2023-08-25T02:36:58.553 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.044492737, 2023-08-25T02:36:58.553 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000042333 2023-08-25T02:36:58.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.553 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:58.553 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 1488000, 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 121280 2023-08-25T02:36:58.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.555 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:36:58.555 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 1324, 2023-08-25T02:36:58.555 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 1473, 2023-08-25T02:36:58.555 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 10994897, 2023-08-25T02:36:58.555 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 829096, 2023-08-25T02:36:58.556 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 7, 2023-08-25T02:36:58.556 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 5, 2023-08-25T02:36:58.556 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.098186528, 2023-08-25T02:36:58.556 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.030678744, 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.027421186, 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.032523391, 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1470, 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.048000131, 2023-08-25T02:36:58.557 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000032653 2023-08-25T02:36:58.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.558 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:58.558 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.558 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.558 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.559 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 320, 2023-08-25T02:36:58.559 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 417888 2023-08-25T02:36:58.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.559 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:36:58.560 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 163, 2023-08-25T02:36:58.560 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 165, 2023-08-25T02:36:58.560 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 8430156, 2023-08-25T02:36:58.560 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 50700, 2023-08-25T02:36:58.560 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 5, 2023-08-25T02:36:58.561 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 4, 2023-08-25T02:36:58.561 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.016259642, 2023-08-25T02:36:58.561 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.003734431, 2023-08-25T02:36:58.561 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.007911948, 2023-08-25T02:36:58.561 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.003528667, 2023-08-25T02:36:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:36:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 163, 2023-08-25T02:36:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.006725459, 2023-08-25T02:36:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000041260 2023-08-25T02:36:58.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:36:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.564 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:36:58.564 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:36:58.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.564 INFO:teuthology.orchestra.run.smithi122.stdout: "cct": { 2023-08-25T02:36:58.564 INFO:teuthology.orchestra.run.smithi122.stdout: "total_workers": 1, 2023-08-25T02:36:58.565 INFO:teuthology.orchestra.run.smithi122.stdout: "unhealthy_workers": 0 2023-08-25T02:36:58.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.565 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-MDSRank": { 2023-08-25T02:36:58.565 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:36:58.566 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:36:58.566 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 3950, 2023-08-25T02:36:58.566 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.114868053, 2023-08-25T02:36:58.566 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000029080 2023-08-25T02:36:58.566 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.567 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-PurgeQueue": { 2023-08-25T02:36:58.567 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:36:58.567 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:36:58.567 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.568 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.568 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.568 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.568 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": { 2023-08-25T02:36:58.569 INFO:teuthology.orchestra.run.smithi122.stdout: "request": 0, 2023-08-25T02:36:58.569 INFO:teuthology.orchestra.run.smithi122.stdout: "reply": 0, 2023-08-25T02:36:58.569 INFO:teuthology.orchestra.run.smithi122.stdout: "reply_latency": { 2023-08-25T02:36:58.569 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.569 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.570 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.570 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_reply": 0, 2023-08-25T02:36:58.570 INFO:teuthology.orchestra.run.smithi122.stdout: "forward": 0, 2023-08-25T02:36:58.570 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_complete": 0, 2023-08-25T02:36:58.571 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_keys": 0, 2023-08-25T02:36:58.571 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_commit": 0, 2023-08-25T02:36:58.571 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_split": 0, 2023-08-25T02:36:58.571 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_merge": 0, 2023-08-25T02:36:58.571 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes": 2460, 2023-08-25T02:36:58.572 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_top": 791, 2023-08-25T02:36:58.572 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_bottom": 1635, 2023-08-25T02:36:58.572 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pin_tail": 34, 2023-08-25T02:36:58.572 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pinned": 1329, 2023-08-25T02:36:58.572 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_expired": 32, 2023-08-25T02:36:58.573 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_with_caps": 0, 2023-08-25T02:36:58.573 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": 0, 2023-08-25T02:36:58.573 INFO:teuthology.orchestra.run.smithi122.stdout: "subtrees": 63, 2023-08-25T02:36:58.573 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse": 0, 2023-08-25T02:36:58.573 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_hit": 0, 2023-08-25T02:36:58.574 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_forward": 0, 2023-08-25T02:36:58.574 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_discover": 0, 2023-08-25T02:36:58.574 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_dir_fetch": 0, 2023-08-25T02:36:58.574 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_remote_ino": 0, 2023-08-25T02:36:58.575 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_lock": 0, 2023-08-25T02:36:58.575 INFO:teuthology.orchestra.run.smithi122.stdout: "load_cent": 0, 2023-08-25T02:36:58.575 INFO:teuthology.orchestra.run.smithi122.stdout: "q": 0, 2023-08-25T02:36:58.576 INFO:teuthology.orchestra.run.smithi122.stdout: "exported": 0, 2023-08-25T02:36:58.576 INFO:teuthology.orchestra.run.smithi122.stdout: "exported_inodes": 0, 2023-08-25T02:36:58.576 INFO:teuthology.orchestra.run.smithi122.stdout: "imported": 0, 2023-08-25T02:36:58.577 INFO:teuthology.orchestra.run.smithi122.stdout: "imported_inodes": 0, 2023-08-25T02:36:58.577 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_dir_fetch": 0, 2023-08-25T02:36:58.578 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:36:58.578 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_peer_discover": 0, 2023-08-25T02:36:58.579 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rfiles": 1044, 2023-08-25T02:36:58.579 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rbytes": 7342262622, 2023-08-25T02:36:58.579 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rsnaps": 6, 2023-08-25T02:36:58.580 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:36:58.580 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_set_tag": 0, 2023-08-25T02:36:58.580 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:36:58.581 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:36:58.581 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:36:58.582 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:36:58.582 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:36:58.582 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_file_inodes": 0, 2023-08-25T02:36:58.583 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_inode_file_caps": 0, 2023-08-25T02:36:58.583 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_revoke": 0, 2023-08-25T02:36:58.583 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_grant": 0, 2023-08-25T02:36:58.584 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:36:58.584 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-08-25T02:36:58.585 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:36:58.585 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps": 0, 2023-08-25T02:36:58.585 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps_dirty": 0, 2023-08-25T02:36:58.586 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_cap_release": 0, 2023-08-25T02:36:58.586 INFO:teuthology.orchestra.run.smithi122.stdout: "process_request_cap_release": 0 2023-08-25T02:36:58.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.587 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_cache": { 2023-08-25T02:36:58.587 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update": 0, 2023-08-25T02:36:58.588 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update_receipt": 0, 2023-08-25T02:36:58.588 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_try_discover": 0, 2023-08-25T02:36:58.589 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_send_discover": 0, 2023-08-25T02:36:58.589 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_handle_discover": 0, 2023-08-25T02:36:58.589 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays": 0, 2023-08-25T02:36:58.590 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_delayed": 0, 2023-08-25T02:36:58.590 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:36:58.591 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_created": 0, 2023-08-25T02:36:58.591 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_enqueued": 0, 2023-08-25T02:36:58.591 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_reintegrated": 0, 2023-08-25T02:36:58.592 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_migrated": 0, 2023-08-25T02:36:58.592 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_processing": 0, 2023-08-25T02:36:58.592 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:36:58.593 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:36:58.593 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_started": 0, 2023-08-25T02:36:58.594 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_completed": 0, 2023-08-25T02:36:58.594 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:36:58.594 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_exportdir": 0, 2023-08-25T02:36:58.595 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_flush": 0, 2023-08-25T02:36:58.595 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragmentdir": 0, 2023-08-25T02:36:58.595 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragstats": 0, 2023-08-25T02:36:58.596 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_inodestats": 0 2023-08-25T02:36:58.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.597 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_log": { 2023-08-25T02:36:58.597 INFO:teuthology.orchestra.run.smithi122.stdout: "evlrg": 0, 2023-08-25T02:36:58.597 INFO:teuthology.orchestra.run.smithi122.stdout: "evadd": 0, 2023-08-25T02:36:58.598 INFO:teuthology.orchestra.run.smithi122.stdout: "evex": 0, 2023-08-25T02:36:58.598 INFO:teuthology.orchestra.run.smithi122.stdout: "evtrm": 0, 2023-08-25T02:36:58.598 INFO:teuthology.orchestra.run.smithi122.stdout: "ev": 8294, 2023-08-25T02:36:58.598 INFO:teuthology.orchestra.run.smithi122.stdout: "evexg": 0, 2023-08-25T02:36:58.598 INFO:teuthology.orchestra.run.smithi122.stdout: "evexd": 0, 2023-08-25T02:36:58.599 INFO:teuthology.orchestra.run.smithi122.stdout: "segadd": 0, 2023-08-25T02:36:58.599 INFO:teuthology.orchestra.run.smithi122.stdout: "segex": 0, 2023-08-25T02:36:58.599 INFO:teuthology.orchestra.run.smithi122.stdout: "segtrm": 0, 2023-08-25T02:36:58.599 INFO:teuthology.orchestra.run.smithi122.stdout: "seg": 8, 2023-08-25T02:36:58.599 INFO:teuthology.orchestra.run.smithi122.stdout: "segmjr": 25, 2023-08-25T02:36:58.600 INFO:teuthology.orchestra.run.smithi122.stdout: "segexg": 0, 2023-08-25T02:36:58.600 INFO:teuthology.orchestra.run.smithi122.stdout: "segexd": 0, 2023-08-25T02:36:58.600 INFO:teuthology.orchestra.run.smithi122.stdout: "expos": 14742073, 2023-08-25T02:36:58.600 INFO:teuthology.orchestra.run.smithi122.stdout: "wrpos": 31631013, 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: "rdpos": 31630901, 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: "jlat": { 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: "replayed": 8294 2023-08-25T02:36:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_mem": { 2023-08-25T02:36:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 1943, 2023-08-25T02:36:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: "ino+": 3083, 2023-08-25T02:36:58.603 INFO:teuthology.orchestra.run.smithi122.stdout: "ino-": 1140, 2023-08-25T02:36:58.603 INFO:teuthology.orchestra.run.smithi122.stdout: "dir": 178, 2023-08-25T02:36:58.603 INFO:teuthology.orchestra.run.smithi122.stdout: "dir+": 204, 2023-08-25T02:36:58.603 INFO:teuthology.orchestra.run.smithi122.stdout: "dir-": 26, 2023-08-25T02:36:58.603 INFO:teuthology.orchestra.run.smithi122.stdout: "dn": 2460, 2023-08-25T02:36:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: "dn+": 3728, 2023-08-25T02:36:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: "dn-": 1268, 2023-08-25T02:36:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: "cap": 0, 2023-08-25T02:36:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: "cap+": 0, 2023-08-25T02:36:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: "cap-": 0, 2023-08-25T02:36:58.605 INFO:teuthology.orchestra.run.smithi122.stdout: "rss": 56132, 2023-08-25T02:36:58.605 INFO:teuthology.orchestra.run.smithi122.stdout: "heap": 207152 2023-08-25T02:36:58.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.605 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_server": { 2023-08-25T02:36:58.606 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_client_request": 0, 2023-08-25T02:36:58.606 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_server_request": 0, 2023-08-25T02:36:58.606 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_request": 0, 2023-08-25T02:36:58.606 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_session": 0, 2023-08-25T02:36:58.606 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_peer_request": 0, 2023-08-25T02:36:58.607 INFO:teuthology.orchestra.run.smithi122.stdout: "req_create_latency": { 2023-08-25T02:36:58.607 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.607 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.607 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.608 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getattr_latency": { 2023-08-25T02:36:58.608 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.608 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.608 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.609 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getfilelock_latency": { 2023-08-25T02:36:58.609 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.609 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.609 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: "req_link_latency": { 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.610 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookup_latency": { 2023-08-25T02:36:58.611 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.611 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.611 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.611 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookuphash_latency": { 2023-08-25T02:36:58.612 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.612 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.612 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.613 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupino_latency": { 2023-08-25T02:36:58.613 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.613 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.613 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.614 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupname_latency": { 2023-08-25T02:36:58.614 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.614 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.614 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.615 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupparent_latency": { 2023-08-25T02:36:58.615 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.615 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.615 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.616 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupsnap_latency": { 2023-08-25T02:36:58.616 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.616 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.616 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.617 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lssnap_latency": { 2023-08-25T02:36:58.617 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.617 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.617 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.618 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mkdir_latency": { 2023-08-25T02:36:58.618 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.618 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.618 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mknod_latency": { 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.620 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mksnap_latency": { 2023-08-25T02:36:58.620 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.620 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.620 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.621 INFO:teuthology.orchestra.run.smithi122.stdout: "req_open_latency": { 2023-08-25T02:36:58.621 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.621 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.621 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.622 INFO:teuthology.orchestra.run.smithi122.stdout: "req_readdir_latency": { 2023-08-25T02:36:58.622 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.622 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.622 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.623 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rename_latency": { 2023-08-25T02:36:58.623 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.623 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.623 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.624 INFO:teuthology.orchestra.run.smithi122.stdout: "req_renamesnap_latency": { 2023-08-25T02:36:58.624 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.624 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.624 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.625 INFO:teuthology.orchestra.run.smithi122.stdout: "req_snapdiff_latency": { 2023-08-25T02:36:58.625 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.625 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.625 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmdir_latency": { 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.627 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmsnap_latency": { 2023-08-25T02:36:58.627 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.627 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.627 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmxattr_latency": { 2023-08-25T02:36:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.629 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setattr_latency": { 2023-08-25T02:36:58.629 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.629 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.629 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.630 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setdirlayout_latency": { 2023-08-25T02:36:58.630 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.630 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.630 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.631 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setfilelock_latency": { 2023-08-25T02:36:58.631 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.631 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.631 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.632 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setlayout_latency": { 2023-08-25T02:36:58.632 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.632 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.632 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.633 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setxattr_latency": { 2023-08-25T02:36:58.633 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.633 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.633 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.634 INFO:teuthology.orchestra.run.smithi122.stdout: "req_symlink_latency": { 2023-08-25T02:36:58.634 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.634 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.634 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.635 INFO:teuthology.orchestra.run.smithi122.stdout: "req_unlink_latency": { 2023-08-25T02:36:58.635 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.635 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.635 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.636 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:36:58.636 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:36:58.636 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getvxattr_latency": { 2023-08-25T02:36:58.636 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.636 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_sessions": { 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: "session_count": 2, 2023-08-25T02:36:58.637 INFO:teuthology.orchestra.run.smithi122.stdout: "session_add": 3, 2023-08-25T02:36:58.638 INFO:teuthology.orchestra.run.smithi122.stdout: "session_remove": 1, 2023-08-25T02:36:58.638 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_open": 3, 2023-08-25T02:36:58.638 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_stale": 0, 2023-08-25T02:36:58.638 INFO:teuthology.orchestra.run.smithi122.stdout: "total_load": 0, 2023-08-25T02:36:58.638 INFO:teuthology.orchestra.run.smithi122.stdout: "average_load": 0, 2023-08-25T02:36:58.639 INFO:teuthology.orchestra.run.smithi122.stdout: "avg_session_uptime": 2838 2023-08-25T02:36:58.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.639 INFO:teuthology.orchestra.run.smithi122.stdout: "mempool": { 2023-08-25T02:36:58.639 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:36:58.640 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_items": 0, 2023-08-25T02:36:58.640 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:36:58.640 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:36:58.640 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:36:58.640 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:36:58.641 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:36:58.641 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:36:58.641 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:36:58.641 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:36:58.641 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:36:58.642 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:36:58.642 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:36:58.642 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:36:58.642 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:36:58.643 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_items": 0, 2023-08-25T02:36:58.643 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:36:58.643 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_items": 0, 2023-08-25T02:36:58.644 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:36:58.644 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:36:58.644 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:36:58.645 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:36:58.645 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:36:58.645 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:36:58.645 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:36:58.646 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_items": 0, 2023-08-25T02:36:58.646 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:36:58.646 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:36:58.646 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:36:58.646 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_items": 0, 2023-08-25T02:36:58.647 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_bytes": 0, 2023-08-25T02:36:58.647 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_items": 0, 2023-08-25T02:36:58.647 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:36:58.647 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:36:58.647 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:36:58.648 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:36:58.648 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_bytes": 277158, 2023-08-25T02:36:58.648 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_items": 2190, 2023-08-25T02:36:58.648 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:36:58.649 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_items": 0, 2023-08-25T02:36:58.649 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_bytes": 0, 2023-08-25T02:36:58.649 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_items": 0, 2023-08-25T02:36:58.649 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:36:58.649 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_items": 0, 2023-08-25T02:36:58.650 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:36:58.650 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_items": 0, 2023-08-25T02:36:58.650 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_bytes": 8224, 2023-08-25T02:36:58.650 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_items": 171, 2023-08-25T02:36:58.650 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:36:58.651 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:36:58.651 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_bytes": 0, 2023-08-25T02:36:58.651 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_items": 0, 2023-08-25T02:36:58.651 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_bytes": 6512983, 2023-08-25T02:36:58.652 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_items": 77543, 2023-08-25T02:36:58.652 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_bytes": 0, 2023-08-25T02:36:58.652 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_items": 0, 2023-08-25T02:36:58.652 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_bytes": 0, 2023-08-25T02:36:58.653 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_items": 0 2023-08-25T02:36:58.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.653 INFO:teuthology.orchestra.run.smithi122.stdout: "objecter": { 2023-08-25T02:36:58.654 INFO:teuthology.orchestra.run.smithi122.stdout: "op_active": 0, 2023-08-25T02:36:58.654 INFO:teuthology.orchestra.run.smithi122.stdout: "op_laggy": 0, 2023-08-25T02:36:58.654 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send": 2320, 2023-08-25T02:36:58.654 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send_bytes": 26, 2023-08-25T02:36:58.655 INFO:teuthology.orchestra.run.smithi122.stdout: "op_resend": 4, 2023-08-25T02:36:58.655 INFO:teuthology.orchestra.run.smithi122.stdout: "op_reply": 2320, 2023-08-25T02:36:58.655 INFO:teuthology.orchestra.run.smithi122.stdout: "op_latency": { 2023-08-25T02:36:58.655 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2320, 2023-08-25T02:36:58.655 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 1.291970362, 2023-08-25T02:36:58.656 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000556883 2023-08-25T02:36:58.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.656 INFO:teuthology.orchestra.run.smithi122.stdout: "op_inflight": 0, 2023-08-25T02:36:58.656 INFO:teuthology.orchestra.run.smithi122.stdout: "oplen_avg": { 2023-08-25T02:36:58.656 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2320, 2023-08-25T02:36:58.657 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 2321 2023-08-25T02:36:58.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.657 INFO:teuthology.orchestra.run.smithi122.stdout: "op": 2320, 2023-08-25T02:36:58.657 INFO:teuthology.orchestra.run.smithi122.stdout: "op_r": 2320, 2023-08-25T02:36:58.657 INFO:teuthology.orchestra.run.smithi122.stdout: "op_w": 0, 2023-08-25T02:36:58.658 INFO:teuthology.orchestra.run.smithi122.stdout: "op_rmw": 0, 2023-08-25T02:36:58.658 INFO:teuthology.orchestra.run.smithi122.stdout: "op_pg": 0, 2023-08-25T02:36:58.658 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_stat": 1372, 2023-08-25T02:36:58.658 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_create": 0, 2023-08-25T02:36:58.658 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_read": 945, 2023-08-25T02:36:58.659 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_write": 0, 2023-08-25T02:36:58.659 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writefull": 0, 2023-08-25T02:36:58.659 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writesame": 0, 2023-08-25T02:36:58.659 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_append": 0, 2023-08-25T02:36:58.659 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_zero": 0, 2023-08-25T02:36:58.660 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_truncate": 0, 2023-08-25T02:36:58.660 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_delete": 0, 2023-08-25T02:36:58.660 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_mapext": 0, 2023-08-25T02:36:58.660 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_sparse_read": 0, 2023-08-25T02:36:58.660 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_clonerange": 0, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_getxattr": 2, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_setxattr": 0, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_rmxattr": 0, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:36:58.661 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_call": 0, 2023-08-25T02:36:58.662 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_watch": 0, 2023-08-25T02:36:58.662 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_notify": 0, 2023-08-25T02:36:58.662 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:36:58.662 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls": 0, 2023-08-25T02:36:58.662 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:36:58.663 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_other": 0, 2023-08-25T02:36:58.663 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_active": 0, 2023-08-25T02:36:58.663 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_send": 0, 2023-08-25T02:36:58.663 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_resend": 0, 2023-08-25T02:36:58.663 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_ping": 0, 2023-08-25T02:36:58.664 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_active": 0, 2023-08-25T02:36:58.664 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_send": 0, 2023-08-25T02:36:58.664 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_resend": 0, 2023-08-25T02:36:58.664 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_active": 0, 2023-08-25T02:36:58.664 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_send": 0, 2023-08-25T02:36:58.665 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_resend": 0, 2023-08-25T02:36:58.665 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_active": 0, 2023-08-25T02:36:58.665 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_send": 0, 2023-08-25T02:36:58.665 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_resend": 0, 2023-08-25T02:36:58.665 INFO:teuthology.orchestra.run.smithi122.stdout: "command_active": 0, 2023-08-25T02:36:58.666 INFO:teuthology.orchestra.run.smithi122.stdout: "command_send": 0, 2023-08-25T02:36:58.666 INFO:teuthology.orchestra.run.smithi122.stdout: "command_resend": 0, 2023-08-25T02:36:58.666 INFO:teuthology.orchestra.run.smithi122.stdout: "map_epoch": 91, 2023-08-25T02:36:58.667 INFO:teuthology.orchestra.run.smithi122.stdout: "map_full": 0, 2023-08-25T02:36:58.667 INFO:teuthology.orchestra.run.smithi122.stdout: "map_inc": 11, 2023-08-25T02:36:58.667 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_sessions": 6, 2023-08-25T02:36:58.667 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_open": 6, 2023-08-25T02:36:58.668 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_close": 0, 2023-08-25T02:36:58.668 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_laggy": 0, 2023-08-25T02:36:58.668 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_wr": 0, 2023-08-25T02:36:58.668 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_rd": 2, 2023-08-25T02:36:58.668 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_del": 0 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: "oft": { 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_objs": 0, 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_updates": 0, 2023-08-25T02:36:58.669 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_removes": 0 2023-08-25T02:36:58.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.670 INFO:teuthology.orchestra.run.smithi122.stdout: "osd-slow-ops": { 2023-08-25T02:36:58.670 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_ops_count": 0 2023-08-25T02:36:58.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.670 INFO:teuthology.orchestra.run.smithi122.stdout: "purge_queue": { 2023-08-25T02:36:58.671 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops": 0, 2023-08-25T02:36:58.671 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops_high_water": 0, 2023-08-25T02:36:58.671 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing": 0, 2023-08-25T02:36:58.671 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_high_water": 0, 2023-08-25T02:36:58.672 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executed": 0, 2023-08-25T02:36:58.672 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_item_in_journal": 0 2023-08-25T02:36:58.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.672 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:36:58.673 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:36:58.673 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:36:58.673 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:36:58.673 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 2623, 2023-08-25T02:36:58.673 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 29320340, 2023-08-25T02:36:58.674 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:58.674 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 2623, 2023-08-25T02:36:58.674 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:36:58.675 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:36:58.675 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2623, 2023-08-25T02:36:58.675 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 29320340, 2023-08-25T02:36:58.676 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:36:58.676 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.677 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.677 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.678 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_bytes": { 2023-08-25T02:36:58.678 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:36:58.679 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:36:58.679 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:36:58.679 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:36:58.679 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:36:58.679 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:58.680 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:58.680 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 2320, 2023-08-25T02:36:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 27437870, 2023-08-25T02:36:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2320, 2023-08-25T02:36:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 27437870, 2023-08-25T02:36:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:36:58.681 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.682 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.682 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.682 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.682 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_ops": { 2023-08-25T02:36:58.683 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:36:58.683 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 1024, 2023-08-25T02:36:58.683 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:36:58.683 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:36:58.683 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:36:58.684 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:58.684 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:58.684 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 2320, 2023-08-25T02:36:58.684 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 2320, 2023-08-25T02:36:58.684 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2320, 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 2320, 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.685 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.686 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:36:58.686 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:36:58.686 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:36:58.686 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:36:58.687 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:36:58.687 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:36:58.687 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:58.687 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:58.687 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:36:58.688 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:36:58.688 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 0, 2023-08-25T02:36:58.688 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 0, 2023-08-25T02:36:58.688 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:36:58.688 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.689 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.689 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.689 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:36:58.689 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle-0x558949538960": { 2023-08-25T02:36:58.690 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:36:58.690 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:36:58.690 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:36:58.690 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:36:58.690 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:36:58.691 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:36:58.691 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:36:58.691 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:36:58.692 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:36:58.692 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 0, 2023-08-25T02:36:58.692 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 0, 2023-08-25T02:36:58.693 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:36:58.693 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:36:58.693 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:36:58.694 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:36:58.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:36:58.695 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-08-25T02:36:58.993 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2023-08-25T02:36:58.993 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2023-08-25T02:36:58.993 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2023-08-25T02:36:58.994 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2023-08-25T02:36:58.994 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2023-08-25T02:36:58.994 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-08-25T02:36:58.994 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-08-25T02:36:58.995 DEBUG:tasks.check_counter:Getting stats from f 2023-08-25T02:36:58.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.f perf dump 2023-08-25T02:36:59.546 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:36:59.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:36:59 smithi122 ceph-mon[100457]: pgmap v325: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:36:59.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:36:59 smithi121 ceph-mon[100099]: pgmap v325: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:37:00.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:36:59 smithi089 ceph-mon[98732]: pgmap v325: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:37:00.876 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-08-25T02:37:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:37:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 2285, 2023-08-25T02:37:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 2284, 2023-08-25T02:37:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 5807794, 2023-08-25T02:37:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 709535, 2023-08-25T02:37:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 6, 2023-08-25T02:37:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 2, 2023-08-25T02:37:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.171545439, 2023-08-25T02:37:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.052457085, 2023-08-25T02:37:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.043070357, 2023-08-25T02:37:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.063066712, 2023-08-25T02:37:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2283, 2023-08-25T02:37:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.063263011, 2023-08-25T02:37:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000027710 2023-08-25T02:37:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-08-25T02:37:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 448 2023-08-25T02:37:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:37:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 4, 2023-08-25T02:37:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 153, 2023-08-25T02:37:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 876, 2023-08-25T02:37:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 418572, 2023-08-25T02:37:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 6, 2023-08-25T02:37:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 4, 2023-08-25T02:37:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.008573928, 2023-08-25T02:37:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.004794700, 2023-08-25T02:37:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.002079856, 2023-08-25T02:37:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.000089248, 2023-08-25T02:37:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 150, 2023-08-25T02:37:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.010459281, 2023-08-25T02:37:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000069728 2023-08-25T02:37:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 320, 2023-08-25T02:37:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 417888 2023-08-25T02:37:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:37:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 302, 2023-08-25T02:37:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 215, 2023-08-25T02:37:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 1485388, 2023-08-25T02:37:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 121607, 2023-08-25T02:37:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 3, 2023-08-25T02:37:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 2, 2023-08-25T02:37:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.039621799, 2023-08-25T02:37:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.012032342, 2023-08-25T02:37:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.030261775, 2023-08-25T02:37:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.004883870, 2023-08-25T02:37:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 214, 2023-08-25T02:37:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.009750261, 2023-08-25T02:37:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000045561 2023-08-25T02:37:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 1485120, 2023-08-25T02:37:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 121296 2023-08-25T02:37:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "cct": { 2023-08-25T02:37:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "total_workers": 1, 2023-08-25T02:37:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "unhealthy_workers": 0 2023-08-25T02:37:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-MDSRank": { 2023-08-25T02:37:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:37:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:37:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 3923, 2023-08-25T02:37:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.092101949, 2023-08-25T02:37:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000023477 2023-08-25T02:37:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-PurgeQueue": { 2023-08-25T02:37:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:37:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:37:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": { 2023-08-25T02:37:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "request": 0, 2023-08-25T02:37:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "reply": 0, 2023-08-25T02:37:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "reply_latency": { 2023-08-25T02:37:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_reply": 0, 2023-08-25T02:37:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "forward": 0, 2023-08-25T02:37:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_complete": 0, 2023-08-25T02:37:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_keys": 0, 2023-08-25T02:37:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_commit": 0, 2023-08-25T02:37:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_split": 0, 2023-08-25T02:37:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_merge": 0, 2023-08-25T02:37:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes": 488, 2023-08-25T02:37:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_top": 54, 2023-08-25T02:37:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_bottom": 256, 2023-08-25T02:37:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pin_tail": 178, 2023-08-25T02:37:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pinned": 410, 2023-08-25T02:37:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_expired": 67, 2023-08-25T02:37:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_with_caps": 0, 2023-08-25T02:37:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": 0, 2023-08-25T02:37:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "subtrees": 16, 2023-08-25T02:37:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse": 0, 2023-08-25T02:37:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_hit": 0, 2023-08-25T02:37:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_forward": 0, 2023-08-25T02:37:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_discover": 0, 2023-08-25T02:37:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_dir_fetch": 0, 2023-08-25T02:37:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_remote_ino": 0, 2023-08-25T02:37:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_lock": 0, 2023-08-25T02:37:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "load_cent": 0, 2023-08-25T02:37:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "q": 0, 2023-08-25T02:37:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "exported": 0, 2023-08-25T02:37:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "exported_inodes": 0, 2023-08-25T02:37:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "imported": 0, 2023-08-25T02:37:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "imported_inodes": 0, 2023-08-25T02:37:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_dir_fetch": 0, 2023-08-25T02:37:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:37:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_peer_discover": 0, 2023-08-25T02:37:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rfiles": 0, 2023-08-25T02:37:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rbytes": 0, 2023-08-25T02:37:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rsnaps": 0, 2023-08-25T02:37:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:37:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_set_tag": 0, 2023-08-25T02:37:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:37:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:37:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:37:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:37:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:37:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_file_inodes": 0, 2023-08-25T02:37:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_inode_file_caps": 0, 2023-08-25T02:37:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_revoke": 0, 2023-08-25T02:37:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_grant": 0, 2023-08-25T02:37:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:37:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-08-25T02:37:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:37:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps": 0, 2023-08-25T02:37:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps_dirty": 0, 2023-08-25T02:37:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_cap_release": 0, 2023-08-25T02:37:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "process_request_cap_release": 0 2023-08-25T02:37:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_cache": { 2023-08-25T02:37:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update": 0, 2023-08-25T02:37:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update_receipt": 0, 2023-08-25T02:37:00.909 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_try_discover": 0, 2023-08-25T02:37:00.909 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_send_discover": 0, 2023-08-25T02:37:00.909 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_handle_discover": 0, 2023-08-25T02:37:00.909 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays": 0, 2023-08-25T02:37:00.909 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_delayed": 0, 2023-08-25T02:37:00.910 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:37:00.910 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_created": 0, 2023-08-25T02:37:00.910 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_enqueued": 0, 2023-08-25T02:37:00.910 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_reintegrated": 0, 2023-08-25T02:37:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_migrated": 0, 2023-08-25T02:37:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_processing": 0, 2023-08-25T02:37:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:37:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:37:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_started": 0, 2023-08-25T02:37:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_completed": 0, 2023-08-25T02:37:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:37:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_exportdir": 0, 2023-08-25T02:37:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_flush": 0, 2023-08-25T02:37:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragmentdir": 0, 2023-08-25T02:37:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragstats": 0, 2023-08-25T02:37:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_inodestats": 0 2023-08-25T02:37:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_log": { 2023-08-25T02:37:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "evlrg": 0, 2023-08-25T02:37:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "evadd": 0, 2023-08-25T02:37:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "evex": 0, 2023-08-25T02:37:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "evtrm": 0, 2023-08-25T02:37:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "ev": 2671, 2023-08-25T02:37:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "evexg": 0, 2023-08-25T02:37:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "evexd": 0, 2023-08-25T02:37:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "segadd": 0, 2023-08-25T02:37:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "segex": 0, 2023-08-25T02:37:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "segtrm": 0, 2023-08-25T02:37:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "seg": 5, 2023-08-25T02:37:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "segmjr": 2, 2023-08-25T02:37:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "segexg": 0, 2023-08-25T02:37:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "segexd": 0, 2023-08-25T02:37:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "expos": 4194304, 2023-08-25T02:37:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "wrpos": 9361283, 2023-08-25T02:37:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "rdpos": 9360709, 2023-08-25T02:37:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "jlat": { 2023-08-25T02:37:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.918 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.918 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.918 INFO:teuthology.orchestra.run.smithi122.stdout: "replayed": 2671 2023-08-25T02:37:00.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.919 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_mem": { 2023-08-25T02:37:00.919 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 272, 2023-08-25T02:37:00.919 INFO:teuthology.orchestra.run.smithi122.stdout: "ino+": 739, 2023-08-25T02:37:00.919 INFO:teuthology.orchestra.run.smithi122.stdout: "ino-": 467, 2023-08-25T02:37:00.919 INFO:teuthology.orchestra.run.smithi122.stdout: "dir": 22, 2023-08-25T02:37:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "dir+": 54, 2023-08-25T02:37:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "dir-": 32, 2023-08-25T02:37:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "dn": 488, 2023-08-25T02:37:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "dn+": 1175, 2023-08-25T02:37:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "dn-": 687, 2023-08-25T02:37:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "cap": 0, 2023-08-25T02:37:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "cap+": 0, 2023-08-25T02:37:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "cap-": 0, 2023-08-25T02:37:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "rss": 52592, 2023-08-25T02:37:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "heap": 207152 2023-08-25T02:37:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_server": { 2023-08-25T02:37:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_client_request": 0, 2023-08-25T02:37:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_server_request": 0, 2023-08-25T02:37:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_request": 0, 2023-08-25T02:37:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_session": 0, 2023-08-25T02:37:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_peer_request": 0, 2023-08-25T02:37:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "req_create_latency": { 2023-08-25T02:37:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getattr_latency": { 2023-08-25T02:37:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getfilelock_latency": { 2023-08-25T02:37:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "req_link_latency": { 2023-08-25T02:37:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookup_latency": { 2023-08-25T02:37:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookuphash_latency": { 2023-08-25T02:37:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupino_latency": { 2023-08-25T02:37:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupname_latency": { 2023-08-25T02:37:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupparent_latency": { 2023-08-25T02:37:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupsnap_latency": { 2023-08-25T02:37:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lssnap_latency": { 2023-08-25T02:37:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mkdir_latency": { 2023-08-25T02:37:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mknod_latency": { 2023-08-25T02:37:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mksnap_latency": { 2023-08-25T02:37:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "req_open_latency": { 2023-08-25T02:37:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "req_readdir_latency": { 2023-08-25T02:37:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rename_latency": { 2023-08-25T02:37:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "req_renamesnap_latency": { 2023-08-25T02:37:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "req_snapdiff_latency": { 2023-08-25T02:37:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmdir_latency": { 2023-08-25T02:37:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmsnap_latency": { 2023-08-25T02:37:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmxattr_latency": { 2023-08-25T02:37:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setattr_latency": { 2023-08-25T02:37:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setdirlayout_latency": { 2023-08-25T02:37:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setfilelock_latency": { 2023-08-25T02:37:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setlayout_latency": { 2023-08-25T02:37:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setxattr_latency": { 2023-08-25T02:37:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "req_symlink_latency": { 2023-08-25T02:37:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "req_unlink_latency": { 2023-08-25T02:37:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:37:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:37:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getvxattr_latency": { 2023-08-25T02:37:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_sessions": { 2023-08-25T02:37:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "session_count": 2, 2023-08-25T02:37:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "session_add": 3, 2023-08-25T02:37:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "session_remove": 1, 2023-08-25T02:37:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_open": 3, 2023-08-25T02:37:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_stale": 0, 2023-08-25T02:37:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "total_load": 0, 2023-08-25T02:37:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "average_load": 0, 2023-08-25T02:37:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "avg_session_uptime": 2836 2023-08-25T02:37:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "mempool": { 2023-08-25T02:37:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:37:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_items": 0, 2023-08-25T02:37:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:37:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:37:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:37:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:37:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:37:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:37:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:37:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:37:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:37:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:37:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:37:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:37:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:37:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_items": 0, 2023-08-25T02:37:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:37:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_items": 0, 2023-08-25T02:37:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:37:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:37:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:37:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:37:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:37:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:37:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:37:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_items": 0, 2023-08-25T02:37:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:37:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:37:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:37:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_items": 0, 2023-08-25T02:37:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_bytes": 0, 2023-08-25T02:37:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_items": 0, 2023-08-25T02:37:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:37:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:37:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:37:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:37:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_bytes": 195649, 2023-08-25T02:37:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_items": 435, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_items": 0, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_bytes": 0, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_items": 0, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:37:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_items": 0, 2023-08-25T02:37:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:37:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_items": 0, 2023-08-25T02:37:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_bytes": 8224, 2023-08-25T02:37:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_items": 171, 2023-08-25T02:37:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:37:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:37:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_bytes": 0, 2023-08-25T02:37:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_items": 0, 2023-08-25T02:37:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_bytes": 1112800, 2023-08-25T02:37:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_items": 20660, 2023-08-25T02:37:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_bytes": 0, 2023-08-25T02:37:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_items": 0, 2023-08-25T02:37:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_bytes": 0, 2023-08-25T02:37:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_items": 0 2023-08-25T02:37:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "objecter": { 2023-08-25T02:37:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "op_active": 0, 2023-08-25T02:37:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "op_laggy": 0, 2023-08-25T02:37:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send": 2283, 2023-08-25T02:37:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send_bytes": 21, 2023-08-25T02:37:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "op_resend": 3, 2023-08-25T02:37:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "op_reply": 2283, 2023-08-25T02:37:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "op_latency": { 2023-08-25T02:37:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2283, 2023-08-25T02:37:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 1.894956552, 2023-08-25T02:37:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000830029 2023-08-25T02:37:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "op_inflight": 0, 2023-08-25T02:37:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "oplen_avg": { 2023-08-25T02:37:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2283, 2023-08-25T02:37:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 2284 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "op": 2283, 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "op_r": 2283, 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "op_w": 0, 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "op_rmw": 0, 2023-08-25T02:37:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "op_pg": 0, 2023-08-25T02:37:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_stat": 1339, 2023-08-25T02:37:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_create": 0, 2023-08-25T02:37:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_read": 942, 2023-08-25T02:37:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_write": 0, 2023-08-25T02:37:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writefull": 0, 2023-08-25T02:37:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writesame": 0, 2023-08-25T02:37:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_append": 0, 2023-08-25T02:37:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_zero": 0, 2023-08-25T02:37:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_truncate": 0, 2023-08-25T02:37:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_delete": 0, 2023-08-25T02:37:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_mapext": 0, 2023-08-25T02:37:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_sparse_read": 0, 2023-08-25T02:37:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_clonerange": 0, 2023-08-25T02:37:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_getxattr": 1, 2023-08-25T02:37:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_setxattr": 0, 2023-08-25T02:37:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:37:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_rmxattr": 0, 2023-08-25T02:37:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:37:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_call": 0, 2023-08-25T02:37:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_watch": 0, 2023-08-25T02:37:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_notify": 0, 2023-08-25T02:37:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:37:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls": 0, 2023-08-25T02:37:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:37:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_other": 0, 2023-08-25T02:37:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_active": 0, 2023-08-25T02:37:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_send": 0, 2023-08-25T02:37:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_resend": 0, 2023-08-25T02:37:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_ping": 0, 2023-08-25T02:37:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_active": 0, 2023-08-25T02:37:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_send": 0, 2023-08-25T02:37:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_resend": 0, 2023-08-25T02:37:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_active": 0, 2023-08-25T02:37:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_send": 0, 2023-08-25T02:37:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_resend": 0, 2023-08-25T02:37:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_active": 0, 2023-08-25T02:37:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_send": 0, 2023-08-25T02:37:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_resend": 0, 2023-08-25T02:37:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "command_active": 0, 2023-08-25T02:37:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "command_send": 0, 2023-08-25T02:37:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "command_resend": 0, 2023-08-25T02:37:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "map_epoch": 91, 2023-08-25T02:37:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "map_full": 0, 2023-08-25T02:37:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "map_inc": 11, 2023-08-25T02:37:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_sessions": 4, 2023-08-25T02:37:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_open": 4, 2023-08-25T02:37:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_close": 0, 2023-08-25T02:37:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_laggy": 0, 2023-08-25T02:37:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_wr": 0, 2023-08-25T02:37:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_rd": 2, 2023-08-25T02:37:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_del": 0 2023-08-25T02:37:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "oft": { 2023-08-25T02:37:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_objs": 0, 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_updates": 0, 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_removes": 0 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "osd-slow-ops": { 2023-08-25T02:37:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_ops_count": 0 2023-08-25T02:37:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "purge_queue": { 2023-08-25T02:37:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops": 0, 2023-08-25T02:37:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops_high_water": 0, 2023-08-25T02:37:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing": 0, 2023-08-25T02:37:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_high_water": 0, 2023-08-25T02:37:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executed": 0, 2023-08-25T02:37:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_item_in_journal": 0 2023-08-25T02:37:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:37:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:37:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 2587, 2023-08-25T02:37:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 7043386, 2023-08-25T02:37:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 2587, 2023-08-25T02:37:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2587, 2023-08-25T02:37:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 7043386, 2023-08-25T02:37:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_bytes": { 2023-08-25T02:37:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:37:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 2283, 2023-08-25T02:37:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 5166984, 2023-08-25T02:37:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2283, 2023-08-25T02:37:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 5166984, 2023-08-25T02:37:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_ops": { 2023-08-25T02:37:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 1024, 2023-08-25T02:37:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 2283, 2023-08-25T02:37:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 2283, 2023-08-25T02:37:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 2283, 2023-08-25T02:37:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 2283, 2023-08-25T02:37:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:37:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:37:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 0, 2023-08-25T02:37:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 0, 2023-08-25T02:37:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle-0x558089146960": { 2023-08-25T02:37:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:37:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 0, 2023-08-25T02:37:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 0, 2023-08-25T02:37:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:01.017 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-08-25T02:37:01.300 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-08-25T02:37:01.301 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-08-25T02:37:01.301 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2023-08-25T02:37:01.301 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2023-08-25T02:37:01.302 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-08-25T02:37:01.302 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-08-25T02:37:01.302 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-08-25T02:37:01.302 DEBUG:tasks.check_counter:Getting stats from i 2023-08-25T02:37:01.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.i perf dump 2023-08-25T02:37:01.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:01 smithi089 ceph-mon[98732]: pgmap v326: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:37:01.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:01 smithi121 ceph-mon[100099]: pgmap v326: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:37:01.728 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:01 smithi122 ceph-mon[100457]: pgmap v326: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-08-25T02:37:01.851 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:37:03.198 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-08-25T02:37:03.198 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:37:03.198 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 13719, 2023-08-25T02:37:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 288058, 2023-08-25T02:37:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 4575987, 2023-08-25T02:37:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 54995696, 2023-08-25T02:37:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 10, 2023-08-25T02:37:03.199 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 5, 2023-08-25T02:37:03.200 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 4.040999211, 2023-08-25T02:37:03.200 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 1.876683338, 2023-08-25T02:37:03.200 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 1.493684344, 2023-08-25T02:37:03.200 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.099045173, 2023-08-25T02:37:03.200 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:03.201 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 288056, 2023-08-25T02:37:03.201 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 2.641527940, 2023-08-25T02:37:03.201 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000009170 2023-08-25T02:37:03.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.201 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:03.202 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 296580, 2023-08-25T02:37:03.202 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.032608152, 2023-08-25T02:37:03.202 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000109 2023-08-25T02:37:03.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.202 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:37:03.203 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:37:03.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.203 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:37:03.203 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 15907, 2023-08-25T02:37:03.203 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 16989, 2023-08-25T02:37:03.204 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 4118848, 2023-08-25T02:37:03.204 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 4453848, 2023-08-25T02:37:03.204 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 11, 2023-08-25T02:37:03.204 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 8, 2023-08-25T02:37:03.204 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.580971245, 2023-08-25T02:37:03.205 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.243432395, 2023-08-25T02:37:03.205 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.399307813, 2023-08-25T02:37:03.205 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.002947763, 2023-08-25T02:37:03.205 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:03.205 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 16986, 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.246023022, 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000014483 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 31763, 2023-08-25T02:37:03.206 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.003799049, 2023-08-25T02:37:03.207 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000119 2023-08-25T02:37:03.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.207 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 1776, 2023-08-25T02:37:03.207 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 116112 2023-08-25T02:37:03.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.208 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:37:03.208 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 447, 2023-08-25T02:37:03.208 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 505, 2023-08-25T02:37:03.208 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 1542672, 2023-08-25T02:37:03.208 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 612878, 2023-08-25T02:37:03.209 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 11, 2023-08-25T02:37:03.209 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 7, 2023-08-25T02:37:03.209 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.054248551, 2023-08-25T02:37:03.209 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.019996237, 2023-08-25T02:37:03.209 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.034697748, 2023-08-25T02:37:03.210 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.006159786, 2023-08-25T02:37:03.210 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:03.210 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 505, 2023-08-25T02:37:03.210 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.028697311, 2023-08-25T02:37:03.210 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000056826 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 38, 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000006990, 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000183 2023-08-25T02:37:03.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.212 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 1507312, 2023-08-25T02:37:03.212 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 506848 2023-08-25T02:37:03.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.212 INFO:teuthology.orchestra.run.smithi122.stdout: "cct": { 2023-08-25T02:37:03.212 INFO:teuthology.orchestra.run.smithi122.stdout: "total_workers": 1, 2023-08-25T02:37:03.213 INFO:teuthology.orchestra.run.smithi122.stdout: "unhealthy_workers": 0 2023-08-25T02:37:03.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.213 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-MDSRank": { 2023-08-25T02:37:03.213 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 2625, 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 89.619199068, 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.034140647 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.215 INFO:teuthology.orchestra.run.smithi122.stdout: "finisher-PurgeQueue": { 2023-08-25T02:37:03.215 INFO:teuthology.orchestra.run.smithi122.stdout: "queue_len": 0, 2023-08-25T02:37:03.215 INFO:teuthology.orchestra.run.smithi122.stdout: "complete_latency": { 2023-08-25T02:37:03.215 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 154, 2023-08-25T02:37:03.215 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 2.666630241, 2023-08-25T02:37:03.216 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.017315780 2023-08-25T02:37:03.216 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.216 INFO:teuthology.orchestra.run.smithi122.stdout: "mds": { 2023-08-25T02:37:03.216 INFO:teuthology.orchestra.run.smithi122.stdout: "request": 5015, 2023-08-25T02:37:03.217 INFO:teuthology.orchestra.run.smithi122.stdout: "reply": 3226, 2023-08-25T02:37:03.217 INFO:teuthology.orchestra.run.smithi122.stdout: "reply_latency": { 2023-08-25T02:37:03.217 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 3226, 2023-08-25T02:37:03.217 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 98799.138717482, 2023-08-25T02:37:03.217 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 30.625895448 2023-08-25T02:37:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_reply": 286, 2023-08-25T02:37:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: "forward": 155, 2023-08-25T02:37:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_complete": 6, 2023-08-25T02:37:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_fetch_keys": 0, 2023-08-25T02:37:03.219 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_commit": 27, 2023-08-25T02:37:03.219 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_split": 2, 2023-08-25T02:37:03.219 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_merge": 0, 2023-08-25T02:37:03.219 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes": 469, 2023-08-25T02:37:03.219 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_top": 133, 2023-08-25T02:37:03.220 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_bottom": 240, 2023-08-25T02:37:03.220 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pin_tail": 96, 2023-08-25T02:37:03.220 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_pinned": 279, 2023-08-25T02:37:03.220 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_expired": 1001, 2023-08-25T02:37:03.220 INFO:teuthology.orchestra.run.smithi122.stdout: "inodes_with_caps": 13, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "caps": 13, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "subtrees": 56, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse": 4240292, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_hit": 3228664, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_forward": 142, 2023-08-25T02:37:03.221 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_discover": 96, 2023-08-25T02:37:03.222 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_dir_fetch": 3, 2023-08-25T02:37:03.222 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_remote_ino": 0, 2023-08-25T02:37:03.222 INFO:teuthology.orchestra.run.smithi122.stdout: "traverse_lock": 981494, 2023-08-25T02:37:03.222 INFO:teuthology.orchestra.run.smithi122.stdout: "load_cent": 1000, 2023-08-25T02:37:03.222 INFO:teuthology.orchestra.run.smithi122.stdout: "q": 1, 2023-08-25T02:37:03.223 INFO:teuthology.orchestra.run.smithi122.stdout: "exported": 26, 2023-08-25T02:37:03.223 INFO:teuthology.orchestra.run.smithi122.stdout: "exported_inodes": 782, 2023-08-25T02:37:03.223 INFO:teuthology.orchestra.run.smithi122.stdout: "imported": 26, 2023-08-25T02:37:03.223 INFO:teuthology.orchestra.run.smithi122.stdout: "imported_inodes": 615, 2023-08-25T02:37:03.223 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_dir_fetch": 0, 2023-08-25T02:37:03.224 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_backtrace_fetch": 0, 2023-08-25T02:37:03.224 INFO:teuthology.orchestra.run.smithi122.stdout: "openino_peer_discover": 104, 2023-08-25T02:37:03.224 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rfiles": 1044, 2023-08-25T02:37:03.224 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rbytes": 7342262210, 2023-08-25T02:37:03.224 INFO:teuthology.orchestra.run.smithi122.stdout: "root_rsnaps": 6, 2023-08-25T02:37:03.225 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_fetch": 0, 2023-08-25T02:37:03.225 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_set_tag": 0, 2023-08-25T02:37:03.225 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_backtrace_repaired": 0, 2023-08-25T02:37:03.225 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_inotable_repaired": 0, 2023-08-25T02:37:03.225 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_inodes": 0, 2023-08-25T02:37:03.226 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dir_base_inodes": 0, 2023-08-25T02:37:03.226 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_dirfrag_rstats": 0, 2023-08-25T02:37:03.226 INFO:teuthology.orchestra.run.smithi122.stdout: "scrub_file_inodes": 0, 2023-08-25T02:37:03.226 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_inode_file_caps": 58, 2023-08-25T02:37:03.226 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_revoke": 489, 2023-08-25T02:37:03.227 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_grant": 974, 2023-08-25T02:37:03.227 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_trunc": 0, 2023-08-25T02:37:03.227 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-08-25T02:37:03.227 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_cap_op_flush_ack": 0, 2023-08-25T02:37:03.227 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps": 1114, 2023-08-25T02:37:03.228 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_caps_dirty": 449, 2023-08-25T02:37:03.228 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_cap_release": 78, 2023-08-25T02:37:03.228 INFO:teuthology.orchestra.run.smithi122.stdout: "process_request_cap_release": 833 2023-08-25T02:37:03.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.228 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_cache": { 2023-08-25T02:37:03.229 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update": 20, 2023-08-25T02:37:03.229 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_update_receipt": 188, 2023-08-25T02:37:03.229 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_try_discover": 111, 2023-08-25T02:37:03.229 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_send_discover": 238, 2023-08-25T02:37:03.229 INFO:teuthology.orchestra.run.smithi122.stdout: "dir_handle_discover": 151, 2023-08-25T02:37:03.230 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays": 2, 2023-08-25T02:37:03.230 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_delayed": 0, 2023-08-25T02:37:03.230 INFO:teuthology.orchestra.run.smithi122.stdout: "num_strays_enqueuing": 0, 2023-08-25T02:37:03.230 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_created": 202, 2023-08-25T02:37:03.230 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_enqueued": 28, 2023-08-25T02:37:03.231 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_reintegrated": 3472, 2023-08-25T02:37:03.231 INFO:teuthology.orchestra.run.smithi122.stdout: "strays_migrated": 18, 2023-08-25T02:37:03.231 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_processing": 0, 2023-08-25T02:37:03.231 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_enqueued": 0, 2023-08-25T02:37:03.232 INFO:teuthology.orchestra.run.smithi122.stdout: "num_recovering_prioritized": 0, 2023-08-25T02:37:03.232 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_started": 0, 2023-08-25T02:37:03.232 INFO:teuthology.orchestra.run.smithi122.stdout: "recovery_completed": 0, 2023-08-25T02:37:03.232 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_enqueue_scrub": 0, 2023-08-25T02:37:03.232 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_exportdir": 26, 2023-08-25T02:37:03.233 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_flush": 0, 2023-08-25T02:37:03.233 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragmentdir": 2, 2023-08-25T02:37:03.233 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_fragstats": 0, 2023-08-25T02:37:03.233 INFO:teuthology.orchestra.run.smithi122.stdout: "ireq_inodestats": 0 2023-08-25T02:37:03.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.234 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_log": { 2023-08-25T02:37:03.234 INFO:teuthology.orchestra.run.smithi122.stdout: "evlrg": 0, 2023-08-25T02:37:03.234 INFO:teuthology.orchestra.run.smithi122.stdout: "evadd": 2671, 2023-08-25T02:37:03.234 INFO:teuthology.orchestra.run.smithi122.stdout: "evex": 0, 2023-08-25T02:37:03.234 INFO:teuthology.orchestra.run.smithi122.stdout: "evtrm": 0, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "ev": 2671, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "evexg": 0, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "evexd": 0, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "segadd": 5, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "segex": 0, 2023-08-25T02:37:03.235 INFO:teuthology.orchestra.run.smithi122.stdout: "segtrm": 0, 2023-08-25T02:37:03.236 INFO:teuthology.orchestra.run.smithi122.stdout: "seg": 5, 2023-08-25T02:37:03.236 INFO:teuthology.orchestra.run.smithi122.stdout: "segmjr": 2, 2023-08-25T02:37:03.236 INFO:teuthology.orchestra.run.smithi122.stdout: "segexg": 0, 2023-08-25T02:37:03.236 INFO:teuthology.orchestra.run.smithi122.stdout: "segexd": 0, 2023-08-25T02:37:03.236 INFO:teuthology.orchestra.run.smithi122.stdout: "expos": 4194304, 2023-08-25T02:37:03.237 INFO:teuthology.orchestra.run.smithi122.stdout: "wrpos": 9361283, 2023-08-25T02:37:03.237 INFO:teuthology.orchestra.run.smithi122.stdout: "rdpos": 0, 2023-08-25T02:37:03.237 INFO:teuthology.orchestra.run.smithi122.stdout: "jlat": { 2023-08-25T02:37:03.237 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1359, 2023-08-25T02:37:03.237 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 57.673659644, 2023-08-25T02:37:03.238 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.042438307 2023-08-25T02:37:03.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.238 INFO:teuthology.orchestra.run.smithi122.stdout: "replayed": 0 2023-08-25T02:37:03.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.238 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_mem": { 2023-08-25T02:37:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: "ino": 310, 2023-08-25T02:37:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: "ino+": 785, 2023-08-25T02:37:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: "ino-": 475, 2023-08-25T02:37:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: "dir": 161, 2023-08-25T02:37:03.239 INFO:teuthology.orchestra.run.smithi122.stdout: "dir+": 228, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "dir-": 67, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "dn": 471, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "dn+": 1570, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "dn-": 1099, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "cap": 13, 2023-08-25T02:37:03.240 INFO:teuthology.orchestra.run.smithi122.stdout: "cap+": 546, 2023-08-25T02:37:03.241 INFO:teuthology.orchestra.run.smithi122.stdout: "cap-": 533, 2023-08-25T02:37:03.241 INFO:teuthology.orchestra.run.smithi122.stdout: "rss": 90984, 2023-08-25T02:37:03.241 INFO:teuthology.orchestra.run.smithi122.stdout: "heap": 198960 2023-08-25T02:37:03.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.242 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_server": { 2023-08-25T02:37:03.242 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_client_request": 2622455, 2023-08-25T02:37:03.242 INFO:teuthology.orchestra.run.smithi122.stdout: "dispatch_server_request": 3690, 2023-08-25T02:37:03.242 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_request": 5015, 2023-08-25T02:37:03.242 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_client_session": 347, 2023-08-25T02:37:03.243 INFO:teuthology.orchestra.run.smithi122.stdout: "handle_peer_request": 10150, 2023-08-25T02:37:03.243 INFO:teuthology.orchestra.run.smithi122.stdout: "req_create_latency": { 2023-08-25T02:37:03.243 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 46, 2023-08-25T02:37:03.243 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 7.898178066, 2023-08-25T02:37:03.243 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.171699523 2023-08-25T02:37:03.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.244 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getattr_latency": { 2023-08-25T02:37:03.244 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 50, 2023-08-25T02:37:03.244 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 8.459557235, 2023-08-25T02:37:03.244 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.169191144 2023-08-25T02:37:03.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.245 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getfilelock_latency": { 2023-08-25T02:37:03.245 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.245 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.245 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.246 INFO:teuthology.orchestra.run.smithi122.stdout: "req_link_latency": { 2023-08-25T02:37:03.246 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 186, 2023-08-25T02:37:03.246 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 50.885503859, 2023-08-25T02:37:03.246 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.273577977 2023-08-25T02:37:03.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.247 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookup_latency": { 2023-08-25T02:37:03.247 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 662, 2023-08-25T02:37:03.247 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 36.558155787, 2023-08-25T02:37:03.247 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.055223800 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookuphash_latency": { 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.249 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupino_latency": { 2023-08-25T02:37:03.249 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.249 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.249 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.250 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupname_latency": { 2023-08-25T02:37:03.250 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.250 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.250 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.251 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupparent_latency": { 2023-08-25T02:37:03.251 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.251 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.251 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lookupsnap_latency": { 2023-08-25T02:37:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.253 INFO:teuthology.orchestra.run.smithi122.stdout: "req_lssnap_latency": { 2023-08-25T02:37:03.253 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.253 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.253 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mkdir_latency": { 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.254 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mknod_latency": { 2023-08-25T02:37:03.255 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.255 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.255 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.255 INFO:teuthology.orchestra.run.smithi122.stdout: "req_mksnap_latency": { 2023-08-25T02:37:03.256 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.256 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.256 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.256 INFO:teuthology.orchestra.run.smithi122.stdout: "req_open_latency": { 2023-08-25T02:37:03.257 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 120, 2023-08-25T02:37:03.257 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 17.047895084, 2023-08-25T02:37:03.257 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.142065792 2023-08-25T02:37:03.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.257 INFO:teuthology.orchestra.run.smithi122.stdout: "req_readdir_latency": { 2023-08-25T02:37:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 80, 2023-08-25T02:37:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 103.215408083, 2023-08-25T02:37:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 1.290192601 2023-08-25T02:37:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.258 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rename_latency": { 2023-08-25T02:37:03.259 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1902, 2023-08-25T02:37:03.259 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 98567.004441380, 2023-08-25T02:37:03.259 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 51.822820421 2023-08-25T02:37:03.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.259 INFO:teuthology.orchestra.run.smithi122.stdout: "req_renamesnap_latency": { 2023-08-25T02:37:03.260 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.260 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.260 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.260 INFO:teuthology.orchestra.run.smithi122.stdout: "req_snapdiff_latency": { 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmdir_latency": { 2023-08-25T02:37:03.261 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.262 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.262 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.262 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmsnap_latency": { 2023-08-25T02:37:03.262 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: "req_rmxattr_latency": { 2023-08-25T02:37:03.263 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.264 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.264 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.264 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setattr_latency": { 2023-08-25T02:37:03.264 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.265 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.265 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.265 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setdirlayout_latency": { 2023-08-25T02:37:03.265 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.266 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.266 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.266 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setfilelock_latency": { 2023-08-25T02:37:03.266 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.267 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.267 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.267 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setlayout_latency": { 2023-08-25T02:37:03.267 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: "req_setxattr_latency": { 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.268 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.269 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.269 INFO:teuthology.orchestra.run.smithi122.stdout: "req_symlink_latency": { 2023-08-25T02:37:03.269 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.269 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.270 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.270 INFO:teuthology.orchestra.run.smithi122.stdout: "req_unlink_latency": { 2023-08-25T02:37:03.270 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 180, 2023-08-25T02:37:03.270 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 8.069577988, 2023-08-25T02:37:03.271 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.044830988 2023-08-25T02:37:03.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.271 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_revoke_eviction": 0, 2023-08-25T02:37:03.271 INFO:teuthology.orchestra.run.smithi122.stdout: "cap_acquisition_throttle": 0, 2023-08-25T02:37:03.271 INFO:teuthology.orchestra.run.smithi122.stdout: "req_getvxattr_latency": { 2023-08-25T02:37:03.272 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.272 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.272 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.272 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.273 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_sessions": { 2023-08-25T02:37:03.273 INFO:teuthology.orchestra.run.smithi122.stdout: "session_count": 2, 2023-08-25T02:37:03.273 INFO:teuthology.orchestra.run.smithi122.stdout: "session_add": 3, 2023-08-25T02:37:03.273 INFO:teuthology.orchestra.run.smithi122.stdout: "session_remove": 1, 2023-08-25T02:37:03.273 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_open": 2, 2023-08-25T02:37:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: "sessions_stale": 0, 2023-08-25T02:37:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: "total_load": 147, 2023-08-25T02:37:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: "average_load": 73, 2023-08-25T02:37:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: "avg_session_uptime": 612 2023-08-25T02:37:03.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "mempool": { 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_items": 0, 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:37:03.275 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:37:03.276 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:37:03.276 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:37:03.276 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:37:03.276 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:37:03.277 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:37:03.277 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:37:03.277 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:37:03.277 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:37:03.277 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:37:03.278 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:37:03.278 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_items": 0, 2023-08-25T02:37:03.278 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:37:03.278 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_items": 0, 2023-08-25T02:37:03.278 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:37:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:37:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:37:03.279 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:37:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:37:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:37:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:37:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_items": 0, 2023-08-25T02:37:03.280 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:37:03.281 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:37:03.281 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:37:03.281 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_items": 0, 2023-08-25T02:37:03.281 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_bytes": 0, 2023-08-25T02:37:03.281 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_items": 0, 2023-08-25T02:37:03.282 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:37:03.282 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:37:03.282 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:37:03.282 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:37:03.282 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_bytes": 861673, 2023-08-25T02:37:03.283 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_items": 574, 2023-08-25T02:37:03.283 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:37:03.283 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_items": 0, 2023-08-25T02:37:03.283 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_bytes": 0, 2023-08-25T02:37:03.283 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_items": 0, 2023-08-25T02:37:03.284 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:37:03.284 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_items": 0, 2023-08-25T02:37:03.284 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:37:03.284 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_items": 0, 2023-08-25T02:37:03.284 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_bytes": 8224, 2023-08-25T02:37:03.285 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_items": 171, 2023-08-25T02:37:03.285 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:37:03.285 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:37:03.285 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_bytes": 0, 2023-08-25T02:37:03.285 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_items": 0, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_bytes": 1449089, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_items": 26819, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_bytes": 0, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_items": 0, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_bytes": 0, 2023-08-25T02:37:03.286 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_items": 0 2023-08-25T02:37:03.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.287 INFO:teuthology.orchestra.run.smithi122.stdout: "objecter": { 2023-08-25T02:37:03.287 INFO:teuthology.orchestra.run.smithi122.stdout: "op_active": 0, 2023-08-25T02:37:03.287 INFO:teuthology.orchestra.run.smithi122.stdout: "op_laggy": 0, 2023-08-25T02:37:03.287 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send": 1621, 2023-08-25T02:37:03.288 INFO:teuthology.orchestra.run.smithi122.stdout: "op_send_bytes": 5401945, 2023-08-25T02:37:03.288 INFO:teuthology.orchestra.run.smithi122.stdout: "op_resend": 27, 2023-08-25T02:37:03.288 INFO:teuthology.orchestra.run.smithi122.stdout: "op_reply": 1621, 2023-08-25T02:37:03.288 INFO:teuthology.orchestra.run.smithi122.stdout: "op_latency": { 2023-08-25T02:37:03.288 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1621, 2023-08-25T02:37:03.289 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 4.837888944, 2023-08-25T02:37:03.289 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.002984508 2023-08-25T02:37:03.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.289 INFO:teuthology.orchestra.run.smithi122.stdout: "op_inflight": 0, 2023-08-25T02:37:03.289 INFO:teuthology.orchestra.run.smithi122.stdout: "oplen_avg": { 2023-08-25T02:37:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 1621, 2023-08-25T02:37:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 1767 2023-08-25T02:37:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: "op": 1621, 2023-08-25T02:37:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: "op_r": 30, 2023-08-25T02:37:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: "op_w": 1591, 2023-08-25T02:37:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: "op_rmw": 0, 2023-08-25T02:37:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: "op_pg": 0, 2023-08-25T02:37:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_stat": 0, 2023-08-25T02:37:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_create": 10, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_read": 24, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_write": 1383, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writefull": 56, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_writesame": 0, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_append": 0, 2023-08-25T02:37:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_zero": 0, 2023-08-25T02:37:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_truncate": 1, 2023-08-25T02:37:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_delete": 42, 2023-08-25T02:37:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_mapext": 0, 2023-08-25T02:37:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_sparse_read": 0, 2023-08-25T02:37:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_clonerange": 0, 2023-08-25T02:37:03.294 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_getxattr": 0, 2023-08-25T02:37:03.294 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_setxattr": 20, 2023-08-25T02:37:03.294 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_cmpxattr": 0, 2023-08-25T02:37:03.294 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_rmxattr": 0, 2023-08-25T02:37:03.294 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_resetxattrs": 0, 2023-08-25T02:37:03.295 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_call": 0, 2023-08-25T02:37:03.295 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_watch": 0, 2023-08-25T02:37:03.295 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_notify": 0, 2023-08-25T02:37:03.295 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_src_cmpxattr": 0, 2023-08-25T02:37:03.295 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls": 0, 2023-08-25T02:37:03.296 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_pgls_filter": 0, 2023-08-25T02:37:03.296 INFO:teuthology.orchestra.run.smithi122.stdout: "osdop_other": 0, 2023-08-25T02:37:03.296 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_active": 0, 2023-08-25T02:37:03.296 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_send": 0, 2023-08-25T02:37:03.296 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_resend": 0, 2023-08-25T02:37:03.297 INFO:teuthology.orchestra.run.smithi122.stdout: "linger_ping": 0, 2023-08-25T02:37:03.297 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_active": 0, 2023-08-25T02:37:03.297 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_send": 0, 2023-08-25T02:37:03.297 INFO:teuthology.orchestra.run.smithi122.stdout: "poolop_resend": 0, 2023-08-25T02:37:03.297 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_active": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_send": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "poolstat_resend": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_active": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_send": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "statfs_resend": 0, 2023-08-25T02:37:03.298 INFO:teuthology.orchestra.run.smithi122.stdout: "command_active": 0, 2023-08-25T02:37:03.299 INFO:teuthology.orchestra.run.smithi122.stdout: "command_send": 0, 2023-08-25T02:37:03.299 INFO:teuthology.orchestra.run.smithi122.stdout: "command_resend": 0, 2023-08-25T02:37:03.299 INFO:teuthology.orchestra.run.smithi122.stdout: "map_epoch": 91, 2023-08-25T02:37:03.299 INFO:teuthology.orchestra.run.smithi122.stdout: "map_full": 0, 2023-08-25T02:37:03.299 INFO:teuthology.orchestra.run.smithi122.stdout: "map_inc": 11, 2023-08-25T02:37:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_sessions": 12, 2023-08-25T02:37:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_open": 12, 2023-08-25T02:37:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_session_close": 0, 2023-08-25T02:37:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_laggy": 0, 2023-08-25T02:37:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_wr": 172, 2023-08-25T02:37:03.301 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_rd": 12, 2023-08-25T02:37:03.301 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_del": 47 2023-08-25T02:37:03.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.301 INFO:teuthology.orchestra.run.smithi122.stdout: "oft": { 2023-08-25T02:37:03.301 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_objs": 1, 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_kv_pairs": 0, 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_updates": 0, 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: "omap_total_removes": 0 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: "osd-slow-ops": { 2023-08-25T02:37:03.302 INFO:teuthology.orchestra.run.smithi122.stdout: "slow_ops_count": 2205 2023-08-25T02:37:03.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.303 INFO:teuthology.orchestra.run.smithi122.stdout: "purge_queue": { 2023-08-25T02:37:03.303 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops": 0, 2023-08-25T02:37:03.303 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_ops_high_water": 3, 2023-08-25T02:37:03.303 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing": 0, 2023-08-25T02:37:03.304 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executing_high_water": 3, 2023-08-25T02:37:03.304 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_executed": 28, 2023-08-25T02:37:03.304 INFO:teuthology.orchestra.run.smithi122.stdout: "pq_item_in_journal": 0 2023-08-25T02:37:03.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.304 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:37:03.305 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:03.305 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:37:03.305 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:03.305 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 30069, 2023-08-25T02:37:03.305 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 7510225, 2023-08-25T02:37:03.306 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:03.306 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 30069, 2023-08-25T02:37:03.306 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:03.306 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:03.306 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 30069, 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 7510225, 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.308 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_bytes": { 2023-08-25T02:37:03.308 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:03.308 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:37:03.308 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:03.309 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:03.309 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:03.309 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:03.309 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:03.309 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 1621, 2023-08-25T02:37:03.310 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 5404797, 2023-08-25T02:37:03.310 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 1621, 2023-08-25T02:37:03.310 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 5404797, 2023-08-25T02:37:03.310 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:03.310 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.311 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-objecter_ops": { 2023-08-25T02:37:03.312 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:03.312 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 1024, 2023-08-25T02:37:03.312 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:03.312 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 0, 2023-08-25T02:37:03.312 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 0, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 0, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 1621, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 1621, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 1621, 2023-08-25T02:37:03.313 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 1621, 2023-08-25T02:37:03.314 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:03.314 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.314 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.314 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.314 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.315 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle": { 2023-08-25T02:37:03.315 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:03.315 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:37:03.315 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:03.316 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 28, 2023-08-25T02:37:03.316 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 2948, 2023-08-25T02:37:03.316 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:03.316 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 28, 2023-08-25T02:37:03.316 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:03.317 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:03.317 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 24, 2023-08-25T02:37:03.317 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 2948, 2023-08-25T02:37:03.317 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:03.317 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.318 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.318 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.318 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:03.318 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-write_buf_throttle-0x55923dd206e0": { 2023-08-25T02:37:03.319 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:03.319 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 3758096384, 2023-08-25T02:37:03.319 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:03.319 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 2671, 2023-08-25T02:37:03.319 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 5166979, 2023-08-25T02:37:03.320 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:03.320 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 2671, 2023-08-25T02:37:03.320 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:03.320 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:03.320 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 1359, 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 5166979, 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:03.321 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.322 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:03.322 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-08-25T02:37:03.606 INFO:tasks.check_counter:Daemon mds.i mds.exported=26 2023-08-25T02:37:03.607 INFO:tasks.check_counter:Daemon mds.i mds.imported=26 2023-08-25T02:37:03.607 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=20 2023-08-25T02:37:03.607 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=188 2023-08-25T02:37:03.607 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-08-25T02:37:03.608 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-08-25T02:37:03.608 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-08-25T02:37:03.608 DEBUG:tasks.check_counter:Getting stats from l 2023-08-25T02:37:03.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba6b39e4f4f77fb95c747099b53db0467d435f56 shell --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c -- ceph daemon mds.l perf dump 2023-08-25T02:37:03.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:03 smithi121 ceph-mon[100099]: pgmap v327: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:03.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:03 smithi122 ceph-mon[100457]: pgmap v327: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:04.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:03 smithi089 ceph-mon[98732]: pgmap v327: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:04.165 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/mon.c/config 2023-08-25T02:37:05.364 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-08-25T02:37:05.364 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-0": { 2023-08-25T02:37:05.365 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 0, 2023-08-25T02:37:05.365 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 2, 2023-08-25T02:37:05.365 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 0, 2023-08-25T02:37:05.365 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 0, 2023-08-25T02:37:05.365 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 4, 2023-08-25T02:37:05.366 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 0, 2023-08-25T02:37:05.366 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.001159954, 2023-08-25T02:37:05.366 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.000000000, 2023-08-25T02:37:05.366 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.000327388, 2023-08-25T02:37:05.367 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:37:05.367 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:05.368 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:05.368 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:05.368 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:05.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.369 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:05.369 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:05.369 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:05.369 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:05.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.370 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 0, 2023-08-25T02:37:05.370 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 0 2023-08-25T02:37:05.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.371 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-1": { 2023-08-25T02:37:05.372 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 210, 2023-08-25T02:37:05.372 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 202, 2023-08-25T02:37:05.372 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 1138992, 2023-08-25T02:37:05.372 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 121024, 2023-08-25T02:37:05.373 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 2, 2023-08-25T02:37:05.373 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 1, 2023-08-25T02:37:05.373 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.036689169, 2023-08-25T02:37:05.373 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.011877194, 2023-08-25T02:37:05.374 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.025488726, 2023-08-25T02:37:05.374 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.005257261, 2023-08-25T02:37:05.375 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:05.375 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 201, 2023-08-25T02:37:05.375 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.008567627, 2023-08-25T02:37:05.376 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000042625 2023-08-25T02:37:05.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.376 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:05.377 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:05.377 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:05.377 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:05.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.378 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 1138992, 2023-08-25T02:37:05.378 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 121024 2023-08-25T02:37:05.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.379 INFO:teuthology.orchestra.run.smithi122.stdout: "AsyncMessenger::Worker-2": { 2023-08-25T02:37:05.379 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_messages": 6, 2023-08-25T02:37:05.379 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages": 156, 2023-08-25T02:37:05.380 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_bytes": 3312, 2023-08-25T02:37:05.380 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_bytes": 179984, 2023-08-25T02:37:05.380 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_created_connections": 5, 2023-08-25T02:37:05.380 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_active_connections": 3, 2023-08-25T02:37:05.380 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_total_time": 0.009404072, 2023-08-25T02:37:05.381 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_send_time": 0.005659898, 2023-08-25T02:37:05.381 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_recv_time": 0.001873005, 2023-08-25T02:37:05.381 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-08-25T02:37:05.382 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_messages_queue_lat": { 2023-08-25T02:37:05.382 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 154, 2023-08-25T02:37:05.382 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.016839874, 2023-08-25T02:37:05.383 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000109349 2023-08-25T02:37:05.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.383 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_handle_ack_lat": { 2023-08-25T02:37:05.383 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:05.384 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:05.384 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:05.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.384 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-08-25T02:37:05.385 INFO:teuthology.orchestra.run.smithi122.stdout: "msgr_send_encrypted_bytes": 179984 2023-08-25T02:37:05.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.385 INFO:teuthology.orchestra.run.smithi122.stdout: "cct": { 2023-08-25T02:37:05.386 INFO:teuthology.orchestra.run.smithi122.stdout: "total_workers": 0, 2023-08-25T02:37:05.386 INFO:teuthology.orchestra.run.smithi122.stdout: "unhealthy_workers": 0 2023-08-25T02:37:05.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.387 INFO:teuthology.orchestra.run.smithi122.stdout: "mempool": { 2023-08-25T02:37:05.387 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_bytes": 0, 2023-08-25T02:37:05.387 INFO:teuthology.orchestra.run.smithi122.stdout: "bloom_filter_items": 0, 2023-08-25T02:37:05.387 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_bytes": 0, 2023-08-25T02:37:05.388 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_alloc_items": 0, 2023-08-25T02:37:05.388 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_bytes": 0, 2023-08-25T02:37:05.388 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_data_items": 0, 2023-08-25T02:37:05.389 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_bytes": 0, 2023-08-25T02:37:05.389 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_onode_items": 0, 2023-08-25T02:37:05.389 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_bytes": 0, 2023-08-25T02:37:05.390 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_meta_items": 0, 2023-08-25T02:37:05.390 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_bytes": 0, 2023-08-25T02:37:05.391 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_other_items": 0, 2023-08-25T02:37:05.391 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_bytes": 0, 2023-08-25T02:37:05.391 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_cache_buffer_items": 0, 2023-08-25T02:37:05.392 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_bytes": 0, 2023-08-25T02:37:05.392 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_extent_items": 0, 2023-08-25T02:37:05.393 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_bytes": 0, 2023-08-25T02:37:05.393 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_blob_items": 0, 2023-08-25T02:37:05.394 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_bytes": 0, 2023-08-25T02:37:05.394 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_shared_blob_items": 0, 2023-08-25T02:37:05.394 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_bytes": 0, 2023-08-25T02:37:05.394 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_inline_bl_items": 0, 2023-08-25T02:37:05.395 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_bytes": 0, 2023-08-25T02:37:05.395 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_fsck_items": 0, 2023-08-25T02:37:05.396 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_bytes": 0, 2023-08-25T02:37:05.396 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_txc_items": 0, 2023-08-25T02:37:05.397 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-25T02:37:05.397 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_deferred_items": 0, 2023-08-25T02:37:05.397 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_bytes": 0, 2023-08-25T02:37:05.397 INFO:teuthology.orchestra.run.smithi122.stdout: "bluestore_writing_items": 0, 2023-08-25T02:37:05.398 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_bytes": 0, 2023-08-25T02:37:05.398 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_items": 0, 2023-08-25T02:37:05.398 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_bytes": 0, 2023-08-25T02:37:05.398 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_reader_items": 0, 2023-08-25T02:37:05.398 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_bytes": 0, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "bluefs_file_writer_items": 0, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_bytes": 149017, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_anon_items": 31, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_bytes": 0, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "buffer_meta_items": 0, 2023-08-25T02:37:05.399 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_bytes": 0, 2023-08-25T02:37:05.400 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_items": 0, 2023-08-25T02:37:05.400 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_bytes": 0, 2023-08-25T02:37:05.400 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_mapbl_items": 0, 2023-08-25T02:37:05.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_bytes": 0, 2023-08-25T02:37:05.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_pglog_items": 0, 2023-08-25T02:37:05.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_bytes": 0, 2023-08-25T02:37:05.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_items": 0, 2023-08-25T02:37:05.401 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_bytes": 0, 2023-08-25T02:37:05.402 INFO:teuthology.orchestra.run.smithi122.stdout: "osdmap_mapping_items": 0, 2023-08-25T02:37:05.402 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_bytes": 0, 2023-08-25T02:37:05.402 INFO:teuthology.orchestra.run.smithi122.stdout: "pgmap_items": 0, 2023-08-25T02:37:05.402 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_bytes": 928, 2023-08-25T02:37:05.402 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_co_items": 2, 2023-08-25T02:37:05.403 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_bytes": 0, 2023-08-25T02:37:05.403 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_1_items": 0, 2023-08-25T02:37:05.403 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_bytes": 0, 2023-08-25T02:37:05.403 INFO:teuthology.orchestra.run.smithi122.stdout: "unittest_2_items": 0 2023-08-25T02:37:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-08-25T02:37:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-08-25T02:37:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "val": 0, 2023-08-25T02:37:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "max": 104857600, 2023-08-25T02:37:05.404 INFO:teuthology.orchestra.run.smithi122.stdout: "get_started": 0, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "get": 212, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "get_sum": 1109377, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_fail": 0, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "get_or_fail_success": 212, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "take": 0, 2023-08-25T02:37:05.405 INFO:teuthology.orchestra.run.smithi122.stdout: "take_sum": 0, 2023-08-25T02:37:05.406 INFO:teuthology.orchestra.run.smithi122.stdout: "put": 212, 2023-08-25T02:37:05.406 INFO:teuthology.orchestra.run.smithi122.stdout: "put_sum": 1109377, 2023-08-25T02:37:05.406 INFO:teuthology.orchestra.run.smithi122.stdout: "wait": { 2023-08-25T02:37:05.406 INFO:teuthology.orchestra.run.smithi122.stdout: "avgcount": 0, 2023-08-25T02:37:05.407 INFO:teuthology.orchestra.run.smithi122.stdout: "sum": 0.000000000, 2023-08-25T02:37:05.407 INFO:teuthology.orchestra.run.smithi122.stdout: "avgtime": 0.000000000 2023-08-25T02:37:05.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:05.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-08-25T02:37:05.408 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-08-25T02:37:05.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:05 smithi122 ceph-mon[100457]: pgmap v328: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:05.712 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-08-25T02:37:05.713 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-08-25T02:37:05.713 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2023-08-25T02:37:05.713 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2023-08-25T02:37:05.714 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-08-25T02:37:05.714 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-08-25T02:37:05.714 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-08-25T02:37:05.715 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-08-25T02:37:05.727 INFO:tasks.kclient:Unmounting kernel clients... 2023-08-25T02:37:05.727 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:37:05.727 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T02:37:05.750 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-08-25T02:37:05.751 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T02:37:05.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:37:05.892 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:05 smithi089 ceph-mon[98732]: pgmap v328: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:05.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:05 smithi121 ceph-mon[100099]: pgmap v328: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:05.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 -- 172.21.15.89:0/1762036516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d08100350 msgr2=0x7f2d0810c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:05.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1762036516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d08100350 0x7f2d0810c6d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000b0a0 tx=0x7f2d0000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:05.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 -- 172.21.15.89:0/1762036516 shutdown_connections 2023-08-25T02:37:05.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1762036516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 0x7f2d08104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:05.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1762036516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d08100350 0x7f2d0810c6d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:05.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1762036516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d080ff9b0 0x7f2d080ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:05.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 -- 172.21.15.89:0/1762036516 >> 172.21.15.89:0/1762036516 conn(0x7f2d080776c0 msgr2=0x7f2d08077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:05.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 -- 172.21.15.89:0/1762036516 shutdown_connections 2023-08-25T02:37:05.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 -- 172.21.15.89:0/1762036516 wait complete. 2023-08-25T02:37:05.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.996+0000 7f2d0f519700 1 Processor -- start 2023-08-25T02:37:05.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 -- start start 2023-08-25T02:37:06.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d08100350 0x7f2d0819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 0x7f2d08195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08195ed0 con 0x7f2d080ff9b0 2023-08-25T02:37:06.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08196040 con 0x7f2d08104850 2023-08-25T02:37:06.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0f519700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08196200 con 0x7f2d08100350 2023-08-25T02:37:06.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49488/0 (socket says 172.21.15.89:49488) 2023-08-25T02:37:06.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 -- 172.21.15.89:0/1269476200 learned_addr learned my addr 172.21.15.89:0/1269476200 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:06.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0cab4700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d08100350 0x7f2d0819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0dab6700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 0x7f2d08195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 -- 172.21.15.89:0/1269476200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d08100350 msgr2=0x7f2d0819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d08100350 0x7f2d0819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 -- 172.21.15.89:0/1269476200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 msgr2=0x7f2d08195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 0x7f2d08195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 -- 172.21.15.89:0/1269476200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009d00 con 0x7f2d080ff9b0 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2d0d2b5700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00d980 tx=0x7f2cfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.997+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc004220 con 0x7f2d080ff9b0 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.998+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cfc007640 con 0x7f2d080ff9b0 2023-08-25T02:37:06.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.998+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081964e0 con 0x7f2d080ff9b0 2023-08-25T02:37:06.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.998+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc005610 con 0x7f2d080ff9b0 2023-08-25T02:37:06.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.998+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d081a27e0 con 0x7f2d080ff9b0 2023-08-25T02:37:06.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:05.998+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d08066e80 con 0x7f2d080ff9b0 2023-08-25T02:37:06.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.000+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f2cfc004750 con 0x7f2d080ff9b0 2023-08-25T02:37:06.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.000+0000 7f2cfa7fc700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 0x7f2cf40a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.000+0000 7f2d0cab4700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 0x7f2cf40a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.000+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f2cfc0cb8d0 con 0x7f2d080ff9b0 2023-08-25T02:37:06.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.001+0000 7f2d0cab4700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 0x7f2cf40a2640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d08077ec0 tx=0x7f2d04009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.004+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f2cfc0952f0 con 0x7f2d080ff9b0 2023-08-25T02:37:06.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.197+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d0810f820 con 0x7f2d080ff9b0 2023-08-25T02:37:06.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.199+0000 7f2cfa7fc700 1 -- 172.21.15.89:0/1269476200 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23043 (secure 0 0 0) 0x7f2cfc016090 con 0x7f2d080ff9b0 2023-08-25T02:37:06.200 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:37:06.201 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:36:15.259886+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":90,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/989947782":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/1951762039":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3424586982":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/2658035648":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:37:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 msgr2=0x7f2cf40a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 0x7f2cf40a2640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2d08077ec0 tx=0x7f2d04009260 comp rx=0 tx=0).stop 2023-08-25T02:37:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 msgr2=0x7f2d0819b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00d980 tx=0x7f2cfc00dc90 comp rx=0 tx=0).stop 2023-08-25T02:37:06.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 shutdown_connections 2023-08-25T02:37:06.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f2cf40a0180 0x7f2cf40a2640 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d08104850 0x7f2d08195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d080ff9b0 0x7f2d0819b7f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 --2- 172.21.15.89:0/1269476200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d08100350 0x7f2d0819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 >> 172.21.15.89:0/1269476200 conn(0x7f2d080776c0 msgr2=0x7f2d081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 shutdown_connections 2023-08-25T02:37:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.201+0000 7f2d0f519700 1 -- 172.21.15.89:0/1269476200 wait complete. 2023-08-25T02:37:06.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T02:21:09.545168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T02:25:02.541375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T02:25:03.205561+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 90, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T02:37:06.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T02:37:06.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 -- 172.21.15.89:0/997681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8100120 msgr2=0x7f59f810c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 --2- 172.21.15.89:0/997681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8100120 0x7f59f810c630 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b0a0 tx=0x7f59f000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 -- 172.21.15.89:0/997681119 shutdown_connections 2023-08-25T02:37:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 --2- 172.21.15.89:0/997681119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f8105770 0x7f59f8105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 --2- 172.21.15.89:0/997681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8100120 0x7f59f810c630 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 --2- 172.21.15.89:0/997681119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f80ff760 0x7f59f80ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 -- 172.21.15.89:0/997681119 >> 172.21.15.89:0/997681119 conn(0x7f59f80fb270 msgr2=0x7f59f80fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:06.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 -- 172.21.15.89:0/997681119 shutdown_connections 2023-08-25T02:37:06.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.378+0000 7f5a00308700 1 -- 172.21.15.89:0/997681119 wait complete. 2023-08-25T02:37:06.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 Processor -- start 2023-08-25T02:37:06.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 -- start start 2023-08-25T02:37:06.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f80ff760 0x7f59f8197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f8100120 0x7f59f8197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f8198580 con 0x7f59f8105770 2023-08-25T02:37:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81a2cc0 con 0x7f59f80ff760 2023-08-25T02:37:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f5a00308700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81a2e80 con 0x7f59f8100120 2023-08-25T02:37:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49506/0 (socket says 172.21.15.89:49506) 2023-08-25T02:37:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 -- 172.21.15.89:0/558859596 learned_addr learned my addr 172.21.15.89:0/558859596 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fd8a3700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f8100120 0x7f59f8197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe0a4700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f80ff760 0x7f59f8197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 -- 172.21.15.89:0/558859596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f8100120 msgr2=0x7f59f8197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f8100120 0x7f59f8197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 -- 172.21.15.89:0/558859596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f80ff760 msgr2=0x7f59f8197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f80ff760 0x7f59f8197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.379+0000 7f59fe8a5700 1 -- 172.21.15.89:0/558859596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f0009d00 con 0x7f59f8105770 2023-08-25T02:37:06.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f59fe8a5700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b830 tx=0x7f59f0009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000e040 con 0x7f59f8105770 2023-08-25T02:37:06.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59f0003c00 con 0x7f59f8105770 2023-08-25T02:37:06.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f81a3100 con 0x7f59f8105770 2023-08-25T02:37:06.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0005590 con 0x7f59f8105770 2023-08-25T02:37:06.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.380+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f81a3530 con 0x7f59f8105770 2023-08-25T02:37:06.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.381+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f80676a0 con 0x7f59f8105770 2023-08-25T02:37:06.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.382+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f59f00056f0 con 0x7f59f8105770 2023-08-25T02:37:06.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.382+0000 7f59ef7fe700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 0x7f59e40a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.383+0000 7f59fe0a4700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 0x7f59e40a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.383+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f59f00cd530 con 0x7f59f8105770 2023-08-25T02:37:06.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.383+0000 7f59fe0a4700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 0x7f59e40a2580 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005950 tx=0x7f59e800b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.386+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f59f0097030 con 0x7f59f8105770 2023-08-25T02:37:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.579+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59f810f7e0 con 0x7f59f8105770 2023-08-25T02:37:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.579+0000 7f59ef7fe700 1 -- 172.21.15.89:0/558859596 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59f0003d70 con 0x7f59f8105770 2023-08-25T02:37:06.581 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-08-25T02:37:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 msgr2=0x7f59e40a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 0x7f59e40a2580 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f59e8005950 tx=0x7f59e800b410 comp rx=0 tx=0).stop 2023-08-25T02:37:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 msgr2=0x7f59f81a2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b830 tx=0x7f59f0009c70 comp rx=0 tx=0).stop 2023-08-25T02:37:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 shutdown_connections 2023-08-25T02:37:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f59e40a00c0 0x7f59e40a2580 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f80ff760 0x7f59f8197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59f8105770 0x7f59f81a2780 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 --2- 172.21.15.89:0/558859596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59f8100120 0x7f59f8197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 >> 172.21.15.89:0/558859596 conn(0x7f59f80fb270 msgr2=0x7f59f80fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 shutdown_connections 2023-08-25T02:37:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.582+0000 7f5a00308700 1 -- 172.21.15.89:0/558859596 wait complete. 2023-08-25T02:37:06.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T02:37:06.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 -- 172.21.15.89:0/4158162582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee810ae60 msgr2=0x7f4ee810d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/4158162582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee810ae60 0x7f4ee810d300 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000b0a0 tx=0x7f4ee000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 -- 172.21.15.89:0/4158162582 shutdown_connections 2023-08-25T02:37:06.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/4158162582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee8101a90 0x7f4ee8101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/4158162582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee810ae60 0x7f4ee810d300 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/4158162582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 0x7f4ee810a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.719+0000 7f4eef0fe700 1 -- 172.21.15.89:0/4158162582 >> 172.21.15.89:0/4158162582 conn(0x7f4ee80fb270 msgr2=0x7f4ee80fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:06.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- 172.21.15.89:0/4158162582 shutdown_connections 2023-08-25T02:37:06.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- 172.21.15.89:0/4158162582 wait complete. 2023-08-25T02:37:06.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 Processor -- start 2023-08-25T02:37:06.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- start start 2023-08-25T02:37:06.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 0x7f4ee80ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 0x7f4ee81a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8100450 con 0x7f4ee8101a90 2023-08-25T02:37:06.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee81a51d0 con 0x7f4ee810ae60 2023-08-25T02:37:06.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eef0fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee81a5310 con 0x7f4ee81023b0 2023-08-25T02:37:06.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eece9a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eece9a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49516/0 (socket says 172.21.15.89:49516) 2023-08-25T02:37:06.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.720+0000 7f4eece9a700 1 -- 172.21.15.89:0/2576513795 learned_addr learned my addr 172.21.15.89:0/2576513795 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:06.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4ee7fff700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 0x7f4ee80ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eed69b700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 0x7f4ee81a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 -- 172.21.15.89:0/2576513795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 msgr2=0x7f4ee80ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 0x7f4ee80ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 -- 172.21.15.89:0/2576513795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 msgr2=0x7f4ee81a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 0x7f4ee81a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 -- 172.21.15.89:0/2576513795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009d00 con 0x7f4ee8101a90 2023-08-25T02:37:06.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eed69b700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 0x7f4ee81a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:37:06.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eece9a700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00e9a0 tx=0x7f4edc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc009790 con 0x7f4ee8101a90 2023-08-25T02:37:06.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee81a5570 con 0x7f4ee8101a90 2023-08-25T02:37:06.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4edc00cbd0 con 0x7f4ee8101a90 2023-08-25T02:37:06.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc018680 con 0x7f4ee8101a90 2023-08-25T02:37:06.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.721+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee81a5aa0 con 0x7f4ee8101a90 2023-08-25T02:37:06.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.722+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee80676a0 con 0x7f4ee8101a90 2023-08-25T02:37:06.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.724+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f4edc00cd40 con 0x7f4ee8101a90 2023-08-25T02:37:06.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.724+0000 7f4ee5ffb700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 0x7f4ed80a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:06.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.725+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f4edc014070 con 0x7f4ee8101a90 2023-08-25T02:37:06.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.725+0000 7f4ee7fff700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 0x7f4ed80a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:06.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.725+0000 7f4ee7fff700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 0x7f4ed80a2580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8100ed0 tx=0x7f4ed4009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:06.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.728+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f4edc0955f0 con 0x7f4ee8101a90 2023-08-25T02:37:06.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.918+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ee8100c10 con 0x7f4ee8101a90 2023-08-25T02:37:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.918+0000 7f4ee5ffb700 1 -- 172.21.15.89:0/2576513795 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4edc098c10 con 0x7f4ee8101a90 2023-08-25T02:37:06.920 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-08-25T02:37:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 msgr2=0x7f4ed80a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 0x7f4ed80a2580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8100ed0 tx=0x7f4ed4009260 comp rx=0 tx=0).stop 2023-08-25T02:37:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 msgr2=0x7f4ee80ff6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00e9a0 tx=0x7f4edc00ecb0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 shutdown_connections 2023-08-25T02:37:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f4ed80a00c0 0x7f4ed80a2580 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee810ae60 0x7f4ee81a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ee8101a90 0x7f4ee80ff6e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 --2- 172.21.15.89:0/2576513795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ee81023b0 0x7f4ee80ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.920+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 >> 172.21.15.89:0/2576513795 conn(0x7f4ee80fb270 msgr2=0x7f4ee810d020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.921+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 shutdown_connections 2023-08-25T02:37:06.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:06.921+0000 7f4eef0fe700 1 -- 172.21.15.89:0/2576513795 wait complete. 2023-08-25T02:37:06.933 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:06 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1269476200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:06.933 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:06 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/558859596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:37:06.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T02:37:06.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:06 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1269476200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:06 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/558859596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:37:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:06 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1269476200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:06.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:06 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/558859596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 -- 172.21.15.89:0/2934302392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 msgr2=0x7f8f9c10c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2934302392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c10c610 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400b0a0 tx=0x7f8f9400b3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 -- 172.21.15.89:0/2934302392 shutdown_connections 2023-08-25T02:37:07.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2934302392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c10c610 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2934302392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c105ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2934302392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 0x7f8f9c100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 -- 172.21.15.89:0/2934302392 >> 172.21.15.89:0/2934302392 conn(0x7f8f9c075ca0 msgr2=0x7f8f9c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:07.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 -- 172.21.15.89:0/2934302392 shutdown_connections 2023-08-25T02:37:07.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.059+0000 7f8fa351c700 1 -- 172.21.15.89:0/2934302392 wait complete. 2023-08-25T02:37:07.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 Processor -- start 2023-08-25T02:37:07.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 -- start start 2023-08-25T02:37:07.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 0x7f8f9c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c1a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c1988d0 con 0x7f8f9c100890 2023-08-25T02:37:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c1a3010 con 0x7f8f9c0ffed0 2023-08-25T02:37:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa351c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c1a31d0 con 0x7f8f9c1057c0 2023-08-25T02:37:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa0ab7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa12b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 0x7f8f9c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa0ab7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49532/0 (socket says 172.21.15.89:49532) 2023-08-25T02:37:07.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.060+0000 7f8fa0ab7700 1 -- 172.21.15.89:0/2175740550 learned_addr learned my addr 172.21.15.89:0/2175740550 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:07.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa1ab9700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c1a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 -- 172.21.15.89:0/2175740550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 msgr2=0x7f8f9c1a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c1a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 -- 172.21.15.89:0/2175740550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 msgr2=0x7f8f9c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 0x7f8f9c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 -- 172.21.15.89:0/2175740550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94009d00 con 0x7f8f9c100890 2023-08-25T02:37:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa1ab9700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c1a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:37:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa0ab7700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d950 tx=0x7f8f9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98009940 con 0x7f8f9c100890 2023-08-25T02:37:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f98010460 con 0x7f8f9c100890 2023-08-25T02:37:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c1a34b0 con 0x7f8f9c100890 2023-08-25T02:37:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800f620 con 0x7f8f9c100890 2023-08-25T02:37:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.061+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c1a3a10 con 0x7f8f9c100890 2023-08-25T02:37:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.062+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c066e10 con 0x7f8f9c100890 2023-08-25T02:37:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.063+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f8f980108d0 con 0x7f8f9c100890 2023-08-25T02:37:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.064+0000 7f8f927fc700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 0x7f8f880a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.064+0000 7f8fa12b8700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 0x7f8f880a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.064+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f8f980cb460 con 0x7f8f9c100890 2023-08-25T02:37:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.064+0000 7f8fa12b8700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 0x7f8f880a2600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c005950 tx=0x7f8f8c00b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.067+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8f98094e80 con 0x7f8f9c100890 2023-08-25T02:37:07.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.258+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f9c10f780 con 0x7f8f9c100890 2023-08-25T02:37:07.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.258+0000 7f8f927fc700 1 -- 172.21.15.89:0/2175740550 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f980160a0 con 0x7f8f9c100890 2023-08-25T02:37:07.260 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-08-25T02:37:07.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 msgr2=0x7f8f880a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 0x7f8f880a2600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c005950 tx=0x7f8f8c00b410 comp rx=0 tx=0).stop 2023-08-25T02:37:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 msgr2=0x7f8f9c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800d950 tx=0x7f8f9800dc60 comp rx=0 tx=0).stop 2023-08-25T02:37:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 shutdown_connections 2023-08-25T02:37:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8f880a0140 0x7f8f880a2600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0ffed0 0x7f8f9c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f9c1057c0 0x7f8f9c1a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 --2- 172.21.15.89:0/2175740550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f9c100890 0x7f8f9c197f90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 >> 172.21.15.89:0/2175740550 conn(0x7f8f9c075ca0 msgr2=0x7f8f9c0fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 shutdown_connections 2023-08-25T02:37:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.260+0000 7f8fa351c700 1 -- 172.21.15.89:0/2175740550 wait complete. 2023-08-25T02:37:07.274 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T02:37:07.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 -- 172.21.15.89:0/96630083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390100890 msgr2=0x7fa390100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 --2- 172.21.15.89:0/96630083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390100890 0x7fa390100d10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009990 tx=0x7fa38c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 -- 172.21.15.89:0/96630083 shutdown_connections 2023-08-25T02:37:07.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 --2- 172.21.15.89:0/96630083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa390101250 0x7fa39010d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 --2- 172.21.15.89:0/96630083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3901068a0 0x7fa390106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 --2- 172.21.15.89:0/96630083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390100890 0x7fa390100d10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 -- 172.21.15.89:0/96630083 >> 172.21.15.89:0/96630083 conn(0x7fa3900fc380 msgr2=0x7fa3900fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:07.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 -- 172.21.15.89:0/96630083 shutdown_connections 2023-08-25T02:37:07.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 -- 172.21.15.89:0/96630083 wait complete. 2023-08-25T02:37:07.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.396+0000 7fa3980a5700 1 Processor -- start 2023-08-25T02:37:07.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 -- start start 2023-08-25T02:37:07.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa390100890 0x7fa39019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 0x7fa390196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390196d90 con 0x7fa390101250 2023-08-25T02:37:07.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390196f00 con 0x7fa3901068a0 2023-08-25T02:37:07.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3901970c0 con 0x7fa390100890 2023-08-25T02:37:07.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49546/0 (socket says 172.21.15.89:49546) 2023-08-25T02:37:07.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 -- 172.21.15.89:0/2670057648 learned_addr learned my addr 172.21.15.89:0/2670057648 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:07.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395e41700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa390100890 0x7fa39019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa396642700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 0x7fa390196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 -- 172.21.15.89:0/2670057648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa390100890 msgr2=0x7fa39019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa390100890 0x7fa39019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 -- 172.21.15.89:0/2670057648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 msgr2=0x7fa390196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 0x7fa390196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 -- 172.21.15.89:0/2670057648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0095f0 con 0x7fa390101250 2023-08-25T02:37:07.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa396642700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 0x7fa390196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:37:07.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa395640700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0095c0 tx=0x7fa38c00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:07.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c01d070 con 0x7fa390101250 2023-08-25T02:37:07.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3901973a0 con 0x7fa390101250 2023-08-25T02:37:07.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa38c00fb50 con 0x7fa390101250 2023-08-25T02:37:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.397+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0173f0 con 0x7fa390101250 2023-08-25T02:37:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.398+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3901a36a0 con 0x7fa390101250 2023-08-25T02:37:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.399+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3900676a0 con 0x7fa390101250 2023-08-25T02:37:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.399+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fa38c00fcc0 con 0x7fa390101250 2023-08-25T02:37:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.400+0000 7fa386ffd700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 0x7fa37c0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.400+0000 7fa395e41700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 0x7fa37c0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.400+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7fa38c0cc740 con 0x7fa390101250 2023-08-25T02:37:07.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.400+0000 7fa395e41700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 0x7fa37c0a2580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa380005950 tx=0x7fa38000b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:07.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.404+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fa38c05ce00 con 0x7fa390101250 2023-08-25T02:37:07.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.609+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa390197c30 con 0x7fa390101250 2023-08-25T02:37:07.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.610+0000 7fa386ffd700 1 -- 172.21.15.89:0/2670057648 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v91) v1 ==== 81+0+943 (secure 0 0 0) 0x7fa38c0961f0 con 0x7fa390101250 2023-08-25T02:37:07.611 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/1949193934 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.612 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/989947782 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.612 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1951762039 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.612 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/3825317465 2023-08-26T02:19:03.221042+0000 2023-08-25T02:37:07.612 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3424586982 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3529687955 2023-08-26T02:19:03.221042+0000 2023-08-25T02:37:07.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/3825317465 2023-08-26T02:19:03.221042+0000 2023-08-25T02:37:07.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2328307501 2023-08-26T02:18:40.017976+0000 2023-08-25T02:37:07.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2658035648 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/730683542 2023-08-26T02:18:40.017976+0000 2023-08-25T02:37:07.614 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3827846667 2023-08-26T02:18:40.017976+0000 2023-08-25T02:37:07.614 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4176087350 2023-08-26T02:18:40.017976+0000 2023-08-25T02:37:07.614 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/1949193934 2023-08-26T02:26:30.360077+0000 2023-08-25T02:37:07.614 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/730683542 2023-08-26T02:18:40.017976+0000 2023-08-25T02:37:07.614 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3259031121 2023-08-26T02:19:03.221042+0000 2023-08-25T02:37:07.615 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1543179786 2023-08-26T02:19:03.221042+0000 2023-08-25T02:37:07.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 msgr2=0x7fa37c0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 0x7fa37c0a2580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa380005950 tx=0x7fa38000b410 comp rx=0 tx=0).stop 2023-08-25T02:37:07.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 msgr2=0x7fa39019cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:07.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0095c0 tx=0x7fa38c00f710 comp rx=0 tx=0).stop 2023-08-25T02:37:07.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 shutdown_connections 2023-08-25T02:37:07.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fa37c0a00c0 0x7fa37c0a2580 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3901068a0 0x7fa390196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa390100890 0x7fa39019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 --2- 172.21.15.89:0/2670057648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa390101250 0x7fa39019cbf0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 >> 172.21.15.89:0/2670057648 conn(0x7fa3900fc380 msgr2=0x7fa3900fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 shutdown_connections 2023-08-25T02:37:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:07.612+0000 7fa3980a5700 1 -- 172.21.15.89:0/2670057648 wait complete. 2023-08-25T02:37:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:listed 16 entries 2023-08-25T02:37:07.626 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-08-25T02:37:07.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:37:07.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /tmp/cephfs 2023-08-25T02:37:07.738 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:37:07.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:37:07.739 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2023-08-25T02:37:07.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:37:07.795 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-08-25T02:37:07.811 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--tmp-cephfs (id: 0) 2023-08-25T02:37:07.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:37:07.812 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--tmp-cephfs 2023-08-25T02:37:07.876 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:07 smithi089 ceph-mon[98732]: pgmap v329: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:07.877 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:07 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2576513795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:07 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2175740550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.878 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:07 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2670057648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-08-25T02:37:07.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T02:37:07.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-08-25T02:37:07.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:07 smithi121 ceph-mon[100099]: pgmap v329: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:07 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2576513795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:07 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2175740550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:07 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2670057648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-08-25T02:37:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:07 smithi122 ceph-mon[100457]: pgmap v329: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-08-25T02:37:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:07 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2576513795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:07 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2175740550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-08-25T02:37:07.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:07 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2670057648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-08-25T02:37:08.024 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-08-25T02:37:08.037 INFO:tasks.cephadm:Teardown begin 2023-08-25T02:37:08.037 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-08-25T02:37:08.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:37:08.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.160+0000 7f2523640700 1 -- 172.21.15.89:0/1551874188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 msgr2=0x7f251c101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- 172.21.15.89:0/1551874188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c101e40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f250c009990 tx=0x7f250c009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- 172.21.15.89:0/1551874188 shutdown_connections 2023-08-25T02:37:08.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- 172.21.15.89:0/1551874188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c10ae80 0x7f251c10d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- 172.21.15.89:0/1551874188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c101e40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- 172.21.15.89:0/1551874188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c102380 0x7f251c10a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- 172.21.15.89:0/1551874188 >> 172.21.15.89:0/1551874188 conn(0x7f251c0fb260 msgr2=0x7f251c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:08.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- 172.21.15.89:0/1551874188 shutdown_connections 2023-08-25T02:37:08.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- 172.21.15.89:0/1551874188 wait complete. 2023-08-25T02:37:08.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 Processor -- start 2023-08-25T02:37:08.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- start start 2023-08-25T02:37:08.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c102380 0x7f251c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c10ae80 0x7f251c196810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c196e80 con 0x7f251c101a60 2023-08-25T02:37:08.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c196ff0 con 0x7f251c102380 2023-08-25T02:37:08.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.161+0000 7f2523640700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c1971b0 con 0x7f251c10ae80 2023-08-25T02:37:08.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49558/0 (socket says 172.21.15.89:49558) 2023-08-25T02:37:08.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 -- 172.21.15.89:0/588071959 learned_addr learned my addr 172.21.15.89:0/588071959 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:08.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f2520bdb700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c102380 0x7f251c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f2521bdd700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c10ae80 0x7f251c196810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 -- 172.21.15.89:0/588071959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c10ae80 msgr2=0x7f251c196810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c10ae80 0x7f251c196810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 -- 172.21.15.89:0/588071959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c102380 msgr2=0x7f251c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c102380 0x7f251c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 -- 172.21.15.89:0/588071959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c0095f0 con 0x7f251c101a60 2023-08-25T02:37:08.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25213dc700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f250c000c00 tx=0x7f250c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c01d070 con 0x7f251c101a60 2023-08-25T02:37:08.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c197430 con 0x7f251c101a60 2023-08-25T02:37:08.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f250c021410 con 0x7f251c101a60 2023-08-25T02:37:08.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c1a3790 con 0x7f251c101a60 2023-08-25T02:37:08.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.162+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c017650 con 0x7f251c101a60 2023-08-25T02:37:08.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.163+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c105f10 con 0x7f251c101a60 2023-08-25T02:37:08.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.164+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f250c0217f0 con 0x7f251c101a60 2023-08-25T02:37:08.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.165+0000 7f25127fc700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 0x7f25080a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.165+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f250c0ccf00 con 0x7f251c101a60 2023-08-25T02:37:08.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.165+0000 7f2520bdb700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 0x7f25080a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.165+0000 7f2520bdb700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 0x7f25080a2640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f251c10c370 tx=0x7f2518009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.168+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f250c096950 con 0x7f251c101a60 2023-08-25T02:37:08.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.402+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251c067b60 con 0x7f251c101a60 2023-08-25T02:37:08.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.403+0000 7f25127fc700 1 -- 172.21.15.89:0/588071959 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+11292 (secure 0 0 0) 0x7f250c099f70 con 0x7f251c101a60 2023-08-25T02:37:08.404 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:37:08.404 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:36:13.606306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":134,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":133,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:37:08.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 msgr2=0x7f25080a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 0x7f25080a2640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f251c10c370 tx=0x7f2518009260 comp rx=0 tx=0).stop 2023-08-25T02:37:08.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 msgr2=0x7f251c19c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f250c000c00 tx=0x7f250c0047b0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 shutdown_connections 2023-08-25T02:37:08.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f25080a0180 0x7f25080a2640 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f251c10ae80 0x7f251c196810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f251c101a60 0x7f251c19c7a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 --2- 172.21.15.89:0/588071959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f251c102380 0x7f251c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 >> 172.21.15.89:0/588071959 conn(0x7f251c0fb260 msgr2=0x7f251c0fbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:08.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 shutdown_connections 2023-08-25T02:37:08.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.405+0000 7f2523640700 1 -- 172.21.15.89:0/588071959 wait complete. 2023-08-25T02:37:08.409 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 101 2023-08-25T02:37:08.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T02:37:08.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 -- 172.21.15.89:0/2623336568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36001057a0 msgr2=0x7f3600105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 --2- 172.21.15.89:0/2623336568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36001057a0 0x7f3600105b80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009990 tx=0x7f35f0009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 -- 172.21.15.89:0/2623336568 shutdown_connections 2023-08-25T02:37:08.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 --2- 172.21.15.89:0/2623336568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3600100150 0x7f360010c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 --2- 172.21.15.89:0/2623336568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36001057a0 0x7f3600105b80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 --2- 172.21.15.89:0/2623336568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36000ff790 0x7f36000ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 -- 172.21.15.89:0/2623336568 >> 172.21.15.89:0/2623336568 conn(0x7f36000fb260 msgr2=0x7f36000fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:08.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 -- 172.21.15.89:0/2623336568 shutdown_connections 2023-08-25T02:37:08.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.539+0000 7f360493d700 1 -- 172.21.15.89:0/2623336568 wait complete. 2023-08-25T02:37:08.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 Processor -- start 2023-08-25T02:37:08.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 -- start start 2023-08-25T02:37:08.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36000ff790 0x7f3600197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 0x7f36001a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36001985d0 con 0x7f3600100150 2023-08-25T02:37:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36001a2d10 con 0x7f36000ff790 2023-08-25T02:37:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f360493d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36001a2ed0 con 0x7f36001057a0 2023-08-25T02:37:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49574/0 (socket says 172.21.15.89:49574) 2023-08-25T02:37:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 -- 172.21.15.89:0/3055518613 learned_addr learned my addr 172.21.15.89:0/3055518613 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fe59c700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36000ff790 0x7f3600197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fed9d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 0x7f36001a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 -- 172.21.15.89:0/3055518613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 msgr2=0x7f36001a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 0x7f36001a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 -- 172.21.15.89:0/3055518613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36000ff790 msgr2=0x7f3600197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36000ff790 0x7f3600197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 -- 172.21.15.89:0/3055518613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00095f0 con 0x7f3600100150 2023-08-25T02:37:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fed9d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 0x7f36001a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:37:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35fdd9b700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f35f400d950 tx=0x7f35f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.540+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4009940 con 0x7f3600100150 2023-08-25T02:37:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.541+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36001a31b0 con 0x7f3600100150 2023-08-25T02:37:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.541+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35f4010460 con 0x7f3600100150 2023-08-25T02:37:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.541+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f400f620 con 0x7f3600100150 2023-08-25T02:37:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.541+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36001a3710 con 0x7f3600100150 2023-08-25T02:37:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.542+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3600067b60 con 0x7f3600100150 2023-08-25T02:37:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.544+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f35f40108d0 con 0x7f3600100150 2023-08-25T02:37:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.544+0000 7f35ef7fe700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 0x7f35e80a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.544+0000 7f35fe59c700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 0x7f35e80a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.544+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f35f40cb4f0 con 0x7f3600100150 2023-08-25T02:37:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.544+0000 7f35fe59c700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 0x7f35e80a25c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b2e0 tx=0x7f35f0005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.548+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f35f4094f10 con 0x7f3600100150 2023-08-25T02:37:08.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.786+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f360010f7b0 con 0x7f3600100150 2023-08-25T02:37:08.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.787+0000 7f35ef7fe700 1 -- 172.21.15.89:0/3055518613 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+11292 (secure 0 0 0) 0x7f35f4098530 con 0x7f3600100150 2023-08-25T02:37:08.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:37:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1349178421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1349178421},{"type":"v1","addr":"172.21.15.89:6835","nonce":1349178421}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/826072159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":826072159},{"type":"v1","addr":"172.21.15.89:6837","nonce":826072159}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/3717526782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":3717526782},{"type":"v1","addr":"172.21.15.89:6839","nonce":3717526782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3065557297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3065557297},{"type":"v1","addr":"172.21.15.89:6841","nonce":3065557297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/1630095418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":1630095418},{"type":"v1","addr":"172.21.15.122:6833","nonce":1630095418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2625413815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2625413815},{"type":"v1","addr":"172.21.15.121:6833","nonce":2625413815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-08-25T02:25:04.024043+0000","modified":"2023-08-25T02:36:13.606306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24475,"mds_2":24470},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/217468698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":217468698},{"type":"v1","addr":"172.21.15.121:6835","nonce":217468698}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3100558972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3100558972},{"type":"v1","addr":"172.21.15.122:6835","nonce":3100558972}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6837/2359799361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2359799361},{"type":"v1","addr":"172.21.15.122:6837","nonce":2359799361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6837/2048790446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":2048790446},{"type":"v1","addr":"172.21.15.121:6837","nonce":2048790446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":134,"addr":"172.21.15.122:6839/3754052688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3754052688},{"type":"v1","addr":"172.21.15.122:6839","nonce":3754052688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":133,"addr":"172.21.15.121:6839/1864669733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":1864669733},{"type":"v1","addr":"172.21.15.121:6839","nonce":1864669733}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T02:37:08.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 msgr2=0x7f35e80a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 0x7f35e80a25c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b2e0 tx=0x7f35f0005fd0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 msgr2=0x7f3600197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f35f400d950 tx=0x7f35f400dc60 comp rx=0 tx=0).stop 2023-08-25T02:37:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 shutdown_connections 2023-08-25T02:37:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f35e80a0100 0x7f35e80a25c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36000ff790 0x7f3600197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36001057a0 0x7f36001a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 --2- 172.21.15.89:0/3055518613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3600100150 0x7f3600197c90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 >> 172.21.15.89:0/3055518613 conn(0x7f36000fb260 msgr2=0x7f36000fb890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 shutdown_connections 2023-08-25T02:37:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.789+0000 7f360493d700 1 -- 172.21.15.89:0/3055518613 wait complete. 2023-08-25T02:37:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 101 2023-08-25T02:37:08.800 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:08 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/588071959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:08.802 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T02:37:08.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 -- 172.21.15.89:0/621325463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8581004d0 msgr2=0x7fd8581008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 --2- 172.21.15.89:0/621325463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8581004d0 0x7fd8581008b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd848009990 tx=0x7fd848009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 -- 172.21.15.89:0/621325463 shutdown_connections 2023-08-25T02:37:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 --2- 172.21.15.89:0/621325463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd858104b00 0x7fd85810c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 --2- 172.21.15.89:0/621325463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8581004d0 0x7fd8581008b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 --2- 172.21.15.89:0/621325463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858100e80 0x7fd858104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 -- 172.21.15.89:0/621325463 >> 172.21.15.89:0/621325463 conn(0x7fd858074a00 msgr2=0x7fd858074e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 -- 172.21.15.89:0/621325463 shutdown_connections 2023-08-25T02:37:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 -- 172.21.15.89:0/621325463 wait complete. 2023-08-25T02:37:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.923+0000 7fd85fe82700 1 Processor -- start 2023-08-25T02:37:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 -- start start 2023-08-25T02:37:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 0x7fd8581955c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858104b00 0x7fd858195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8581a5210 con 0x7fd858196040 2023-08-25T02:37:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8581a5350 con 0x7fd8581004d0 2023-08-25T02:37:08.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85fe82700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8581a5510 con 0x7fd858104b00 2023-08-25T02:37:08.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85dc1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 0x7fd8581955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85dc1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 0x7fd8581955c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.89:39996/0 (socket says 172.21.15.89:39996) 2023-08-25T02:37:08.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85dc1e700 1 -- 172.21.15.89:0/1841779255 learned_addr learned my addr 172.21.15.89:0/1841779255 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:08.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49590/0 (socket says 172.21.15.89:49590) 2023-08-25T02:37:08.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85d41d700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858104b00 0x7fd858195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 -- 172.21.15.89:0/1841779255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858104b00 msgr2=0x7fd858195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858104b00 0x7fd858195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 -- 172.21.15.89:0/1841779255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 msgr2=0x7fd8581955c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 0x7fd8581955c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 -- 172.21.15.89:0/1841779255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480095f0 con 0x7fd858196040 2023-08-25T02:37:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd85e41f700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd850007f10 tx=0x7fd85000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850009ea0 con 0x7fd858196040 2023-08-25T02:37:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85000f040 con 0x7fd858196040 2023-08-25T02:37:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.924+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850004350 con 0x7fd858196040 2023-08-25T02:37:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.925+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8581a57f0 con 0x7fd858196040 2023-08-25T02:37:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.925+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8581a5cb0 con 0x7fd858196040 2023-08-25T02:37:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.925+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd858066e80 con 0x7fd858196040 2023-08-25T02:37:08.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.926+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7fd85000db20 con 0x7fd858196040 2023-08-25T02:37:08.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.927+0000 7fd84effd700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 0x7fd8440a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:08.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.927+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7fd8500caf30 con 0x7fd858196040 2023-08-25T02:37:08.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.927+0000 7fd85dc1e700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 0x7fd8440a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:08.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.927+0000 7fd85dc1e700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 0x7fd8440a25c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd848006010 tx=0x7fd84800b260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:08.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:08.931+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7fd850094950 con 0x7fd858196040 2023-08-25T02:37:08.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:08 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/588071959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:08.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:08 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/588071959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.125+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd858101000 con 0x7fd858196040 2023-08-25T02:37:09.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.126+0000 7fd84effd700 1 -- 172.21.15.89:0/1841779255 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+23043 (secure 0 0 0) 0x7fd850019090 con 0x7fd858196040 2023-08-25T02:37:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-08-25T02:37:09.128 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"fsid":"7eb69934-42ed-11ee-9b3d-001a4aab830c","created":"2023-08-25T02:18:08.578395+0000","modified":"2023-08-25T02:36:15.259886+0000","last_up_change":"2023-08-25T02:24:23.355501+0000","last_in_change":"2023-08-25T02:24:05.124805+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T02:21:09.545168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-08-25T02:25:02.541375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-08-25T02:25:03.205561+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":90,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94583d02-1254-4da8-aae0-95aebc0f9804","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6803","nonce":729401604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6805","nonce":729401604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6809","nonce":729401604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":729401604},{"type":"v1","addr":"172.21.15.89:6807","nonce":729401604}]},"public_addr":"172.21.15.89:6803/729401604","cluster_addr":"172.21.15.89:6805/729401604","heartbeat_back_addr":"172.21.15.89:6809/729401604","heartbeat_front_addr":"172.21.15.89:6807/729401604","state":["exists","up"]},{"osd":1,"uuid":"6112d305-65d5-4bdd-b560-8baba3ca0e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6801","nonce":1465063129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6803","nonce":1465063129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6807","nonce":1465063129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1465063129},{"type":"v1","addr":"172.21.15.121:6805","nonce":1465063129}]},"public_addr":"172.21.15.121:6801/1465063129","cluster_addr":"172.21.15.121:6803/1465063129","heartbeat_back_addr":"172.21.15.121:6807/1465063129","heartbeat_front_addr":"172.21.15.121:6805/1465063129","state":["exists","up"]},{"osd":2,"uuid":"e8a1640c-fce8-4ac8-a53f-857213701817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6801","nonce":1354963983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6803","nonce":1354963983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6807","nonce":1354963983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1354963983},{"type":"v1","addr":"172.21.15.122:6805","nonce":1354963983}]},"public_addr":"172.21.15.122:6801/1354963983","cluster_addr":"172.21.15.122:6803/1354963983","heartbeat_back_addr":"172.21.15.122:6807/1354963983","heartbeat_front_addr":"172.21.15.122:6805/1354963983","state":["exists","up"]},{"osd":3,"uuid":"a3c51d36-56d1-4998-b4e3-0c9ba5ef09ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6811","nonce":705728932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6813","nonce":705728932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6817","nonce":705728932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":705728932},{"type":"v1","addr":"172.21.15.89:6815","nonce":705728932}]},"public_addr":"172.21.15.89:6811/705728932","cluster_addr":"172.21.15.89:6813/705728932","heartbeat_back_addr":"172.21.15.89:6817/705728932","heartbeat_front_addr":"172.21.15.89:6815/705728932","state":["exists","up"]},{"osd":4,"uuid":"f57ae45d-1fb5-41e6-9c04-ba9c235814af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6809","nonce":4228849553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6811","nonce":4228849553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6815","nonce":4228849553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":4228849553},{"type":"v1","addr":"172.21.15.121:6813","nonce":4228849553}]},"public_addr":"172.21.15.121:6809/4228849553","cluster_addr":"172.21.15.121:6811/4228849553","heartbeat_back_addr":"172.21.15.121:6815/4228849553","heartbeat_front_addr":"172.21.15.121:6813/4228849553","state":["exists","up"]},{"osd":5,"uuid":"a9751b06-a1ca-4fe3-ba70-2dc2733e05b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6809","nonce":1949734782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6811","nonce":1949734782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6815","nonce":1949734782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1949734782},{"type":"v1","addr":"172.21.15.122:6813","nonce":1949734782}]},"public_addr":"172.21.15.122:6809/1949734782","cluster_addr":"172.21.15.122:6811/1949734782","heartbeat_back_addr":"172.21.15.122:6815/1949734782","heartbeat_front_addr":"172.21.15.122:6813/1949734782","state":["exists","up"]},{"osd":6,"uuid":"a77199fb-6a0a-4a5f-83a1-690b403a5791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6819","nonce":1372241573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6821","nonce":1372241573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6825","nonce":1372241573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1372241573},{"type":"v1","addr":"172.21.15.89:6823","nonce":1372241573}]},"public_addr":"172.21.15.89:6819/1372241573","cluster_addr":"172.21.15.89:6821/1372241573","heartbeat_back_addr":"172.21.15.89:6825/1372241573","heartbeat_front_addr":"172.21.15.89:6823/1372241573","state":["exists","up"]},{"osd":7,"uuid":"f5f512d9-eebe-40fa-beb8-f1e78c6d0af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6817","nonce":62965695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6819","nonce":62965695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6823","nonce":62965695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":62965695},{"type":"v1","addr":"172.21.15.121:6821","nonce":62965695}]},"public_addr":"172.21.15.121:6817/62965695","cluster_addr":"172.21.15.121:6819/62965695","heartbeat_back_addr":"172.21.15.121:6823/62965695","heartbeat_front_addr":"172.21.15.121:6821/62965695","state":["exists","up"]},{"osd":8,"uuid":"3417d93a-10fe-4919-9d05-41ef7869e207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6817","nonce":1782298471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6819","nonce":1782298471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6823","nonce":1782298471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1782298471},{"type":"v1","addr":"172.21.15.122:6821","nonce":1782298471}]},"public_addr":"172.21.15.122:6817/1782298471","cluster_addr":"172.21.15.122:6819/1782298471","heartbeat_back_addr":"172.21.15.122:6823/1782298471","heartbeat_front_addr":"172.21.15.122:6821/1782298471","state":["exists","up"]},{"osd":9,"uuid":"0c4d98e8-6bcc-4ca6-9fec-495733ba95d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6827","nonce":3342522203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6829","nonce":3342522203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6833","nonce":3342522203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":3342522203},{"type":"v1","addr":"172.21.15.89:6831","nonce":3342522203}]},"public_addr":"172.21.15.89:6827/3342522203","cluster_addr":"172.21.15.89:6829/3342522203","heartbeat_back_addr":"172.21.15.89:6833/3342522203","heartbeat_front_addr":"172.21.15.89:6831/3342522203","state":["exists","up"]},{"osd":10,"uuid":"8b4dd502-8b3b-4c22-b3c3-79d177e32289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6825","nonce":3780412782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6827","nonce":3780412782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6831","nonce":3780412782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3780412782},{"type":"v1","addr":"172.21.15.121:6829","nonce":3780412782}]},"public_addr":"172.21.15.121:6825/3780412782","cluster_addr":"172.21.15.121:6827/3780412782","heartbeat_back_addr":"172.21.15.121:6831/3780412782","heartbeat_front_addr":"172.21.15.121:6829/3780412782","state":["exists","up"]},{"osd":11,"uuid":"7e7ae39d-a06b-4a45-9bfd-b969620b53e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6825","nonce":2395101680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6827","nonce":2395101680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6831","nonce":2395101680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2395101680},{"type":"v1","addr":"172.21.15.122:6829","nonce":2395101680}]},"public_addr":"172.21.15.122:6825/2395101680","cluster_addr":"172.21.15.122:6827/2395101680","heartbeat_back_addr":"172.21.15.122:6831/2395101680","heartbeat_front_addr":"172.21.15.122:6829/2395101680","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:24.113676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:20:45.355121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:07.690219+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:29.200134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:21:50.335496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:12.355254+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:33.317043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:22:54.711779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:16.655147+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:38.018963+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:23:59.458921+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T02:24:21.788539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/989947782":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/1951762039":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/3424586982":"2023-08-26T02:26:30.360077+0000","172.21.15.89:0/3529687955":"2023-08-26T02:19:03.221042+0000","172.21.15.89:6800/3825317465":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/2328307501":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/2658035648":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6801/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3827846667":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/4176087350":"2023-08-26T02:18:40.017976+0000","172.21.15.89:6800/1949193934":"2023-08-26T02:26:30.360077+0000","172.21.15.89:6800/730683542":"2023-08-26T02:18:40.017976+0000","172.21.15.89:0/3259031121":"2023-08-26T02:19:03.221042+0000","172.21.15.89:0/1543179786":"2023-08-26T02:19:03.221042+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T02:37:09.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.128+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 msgr2=0x7fd8440a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 0x7fd8440a25c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd848006010 tx=0x7fd84800b260 comp rx=0 tx=0).stop 2023-08-25T02:37:09.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 msgr2=0x7fd8581a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd850007f10 tx=0x7fd85000d3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 shutdown_connections 2023-08-25T02:37:09.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7fd8440a0100 0x7fd8440a25c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8581004d0 0x7fd8581955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd858196040 0x7fd8581a4c40 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 --2- 172.21.15.89:0/1841779255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd858104b00 0x7fd858195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 >> 172.21.15.89:0/1841779255 conn(0x7fd858074a00 msgr2=0x7fd8580fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:09.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 shutdown_connections 2023-08-25T02:37:09.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.129+0000 7fd85fe82700 1 -- 172.21.15.89:0/1841779255 wait complete. 2023-08-25T02:37:09.142 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-08-25T02:37:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 -- 172.21.15.89:0/3977208387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 msgr2=0x7f0a70101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3977208387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a70101eb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009990 tx=0x7f0a64009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 -- 172.21.15.89:0/3977208387 shutdown_connections 2023-08-25T02:37:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3977208387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 0x7f0a7010d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3977208387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a70101eb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3977208387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7010a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 -- 172.21.15.89:0/3977208387 >> 172.21.15.89:0/3977208387 conn(0x7f0a700fb270 msgr2=0x7f0a700fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 -- 172.21.15.89:0/3977208387 shutdown_connections 2023-08-25T02:37:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.264+0000 7f0a7524a700 1 -- 172.21.15.89:0/3977208387 wait complete. 2023-08-25T02:37:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 Processor -- start 2023-08-25T02:37:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 -- start start 2023-08-25T02:37:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 0x7f0a70196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70196df0 con 0x7f0a70101ad0 2023-08-25T02:37:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70196f60 con 0x7f0a701023f0 2023-08-25T02:37:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a7524a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70197120 con 0x7f0a7010aea0 2023-08-25T02:37:09.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a6ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a6ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49600/0 (socket says 172.21.15.89:49600) 2023-08-25T02:37:09.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.265+0000 7f0a6ed9d700 1 -- 172.21.15.89:0/3593818426 learned_addr learned my addr 172.21.15.89:0/3593818426 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:09.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6f59e700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 0x7f0a70196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6e59c700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 -- 172.21.15.89:0/3593818426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 msgr2=0x7f0a70196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 0x7f0a70196780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 -- 172.21.15.89:0/3593818426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 msgr2=0x7f0a7019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 -- 172.21.15.89:0/3593818426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640095f0 con 0x7f0a70101ad0 2023-08-25T02:37:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6e59c700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:37:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a6ed9d700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0a640095c0 tx=0x7f0a6400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6401d070 con 0x7f0a70101ad0 2023-08-25T02:37:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a640054f0 con 0x7f0a70101ad0 2023-08-25T02:37:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a701973a0 con 0x7f0a70101ad0 2023-08-25T02:37:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64004bd0 con 0x7f0a70101ad0 2023-08-25T02:37:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.266+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a701a3700 con 0x7f0a70101ad0 2023-08-25T02:37:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.267+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a70105730 con 0x7f0a70101ad0 2023-08-25T02:37:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.268+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f0a6400fc30 con 0x7f0a70101ad0 2023-08-25T02:37:09.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.269+0000 7f0a57fff700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 0x7f0a580a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.269+0000 7f0a6e59c700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 0x7f0a580a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.269+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8571+0+0 (secure 0 0 0) 0x7f0a640cca60 con 0x7f0a70101ad0 2023-08-25T02:37:09.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.269+0000 7f0a6e59c700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 0x7f0a580a2580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a60005fd0 tx=0x7f0a60005e20 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:09.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.272+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f0a64096510 con 0x7f0a70101ad0 2023-08-25T02:37:09.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.508+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0a700676a0 con 0x7f0a70101ad0 2023-08-25T02:37:09.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.515+0000 7f0a57fff700 1 -- 172.21.15.89:0/3593818426 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0a64099b30 con 0x7f0a70101ad0 2023-08-25T02:37:09.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 msgr2=0x7f0a580a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 0x7f0a580a2580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a60005fd0 tx=0x7f0a60005e20 comp rx=0 tx=0).stop 2023-08-25T02:37:09.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 msgr2=0x7f0a7019c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0a640095c0 tx=0x7f0a6400f740 comp rx=0 tx=0).stop 2023-08-25T02:37:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 shutdown_connections 2023-08-25T02:37:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f0a580a00c0 0x7f0a580a2580 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a7010aea0 0x7f0a70196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a70101ad0 0x7f0a7019c710 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 --2- 172.21.15.89:0/3593818426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a701023f0 0x7f0a7019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 >> 172.21.15.89:0/3593818426 conn(0x7f0a700fb270 msgr2=0x7f0a70102870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:09.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 shutdown_connections 2023-08-25T02:37:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.518+0000 7f0a7524a700 1 -- 172.21.15.89:0/3593818426 wait complete. 2023-08-25T02:37:09.525 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-08-25T02:37:09.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-08-25T02:37:09.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 -- 172.21.15.89:0/1481621376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8100120 msgr2=0x7f67e810c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1481621376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8100120 0x7f67e810c630 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f67e000b0a0 tx=0x7f67e000b3b0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 -- 172.21.15.89:0/1481621376 shutdown_connections 2023-08-25T02:37:09.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1481621376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8105770 0x7f67e8105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1481621376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8100120 0x7f67e810c630 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1481621376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e80ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 -- 172.21.15.89:0/1481621376 >> 172.21.15.89:0/1481621376 conn(0x7f67e80fb270 msgr2=0x7f67e80fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:09.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 -- 172.21.15.89:0/1481621376 shutdown_connections 2023-08-25T02:37:09.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.675+0000 7f67f07ce700 1 -- 172.21.15.89:0/1481621376 wait complete. 2023-08-25T02:37:09.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 Processor -- start 2023-08-25T02:37:09.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 -- start start 2023-08-25T02:37:09.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e81976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 0x7f67e8197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e81a2ca0 con 0x7f67e8105770 2023-08-25T02:37:09.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e81a2de0 con 0x7f67e8100120 2023-08-25T02:37:09.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67f07ce700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e81a2f20 con 0x7f67e80ff760 2023-08-25T02:37:09.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49616/0 (socket says 172.21.15.89:49616) 2023-08-25T02:37:09.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 -- 172.21.15.89:0/1920475775 learned_addr learned my addr 172.21.15.89:0/1920475775 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:09.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67edd69700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 0x7f67e8197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67ee56a700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e81976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 -- 172.21.15.89:0/1920475775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 msgr2=0x7f67e81976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e81976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 -- 172.21.15.89:0/1920475775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 msgr2=0x7f67e8197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:09.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 0x7f67e8197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:09.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67eed6b700 1 -- 172.21.15.89:0/1920475775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e0009d00 con 0x7f67e8105770 2023-08-25T02:37:09.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67edd69700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 0x7f67e8197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T02:37:09.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.676+0000 7f67ee56a700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e81976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T02:37:09.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67eed6b700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f67e000b780 tx=0x7f67e00076f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:09.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e000e040 con 0x7f67e8105770 2023-08-25T02:37:09.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67e00079d0 con 0x7f67e8105770 2023-08-25T02:37:09.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e81a3120 con 0x7f67e8105770 2023-08-25T02:37:09.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0005350 con 0x7f67e8105770 2023-08-25T02:37:09.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.677+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e81a35a0 con 0x7f67e8105770 2023-08-25T02:37:09.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.678+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e80676a0 con 0x7f67e8105770 2023-08-25T02:37:09.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.679+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f67e001da30 con 0x7f67e8105770 2023-08-25T02:37:09.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.679+0000 7f67df7fe700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 0x7f67d40a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:09.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.679+0000 7f67ee56a700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 0x7f67d40a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:09.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.679+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f67e00cd650 con 0x7f67e8105770 2023-08-25T02:37:09.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.680+0000 7f67ee56a700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 0x7f67d40a2580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009bc0 tx=0x7f67d8009260 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:09.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.683+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f67e0096fd0 con 0x7f67e8105770 2023-08-25T02:37:09.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a[98709]: 2023-08-25T02:37:09.509+0000 7ff7cbcb7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-08-25T02:37:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: pgmap v330: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3055518613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1841779255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-08-25T02:37:09.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-08-25T02:37:09.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-08-25T02:37:09.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: osdmap e92: 12 total, 12 up, 12 in 2023-08-25T02:37:09.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-08-25T02:37:09.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:09 smithi089 ceph-mon[98732]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-08-25T02:37:09.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:09.916+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f67e810f7e0 con 0x7f67e8105770 2023-08-25T02:37:09.968 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.969 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.969 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mds[125295]: -- 172.21.15.121:0/195289350 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bbd20b71e0 con 0x55bbd208cc00 2023-08-25T02:37:09.969 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: starting mds.b at 2023-08-25T02:37:09.970 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.970 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.971 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mds[124615]: -- 172.21.15.121:0/1031270301 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d890c751e0 con 0x55d890c4ac00 2023-08-25T02:37:09.971 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: starting mds.e at 2023-08-25T02:37:09.971 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.972 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.972 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mds[124956]: -- 172.21.15.121:0/2304957310 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ef60dd1e0 con 0x562ef60b3000 2023-08-25T02:37:09.972 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: starting mds.h at 2023-08-25T02:37:09.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: pgmap v330: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:09.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3055518613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1841779255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-08-25T02:37:09.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-08-25T02:37:09.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-08-25T02:37:09.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: osdmap e92: 12 total, 12 up, 12 in 2023-08-25T02:37:09.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-08-25T02:37:09.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:09 smithi121 ceph-mon[100099]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-08-25T02:37:09.978 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.978 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.978 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mds[126286]: -- 172.21.15.122:0/626020848 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558582e431e0 con 0x558582e18c00 2023-08-25T02:37:09.979 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: starting mds.c at 2023-08-25T02:37:09.979 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.979 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.979 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: starting mds.f at 2023-08-25T02:37:09.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: pgmap v330: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-08-25T02:37:09.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3055518613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1841779255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-08-25T02:37:09.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-08-25T02:37:09.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-08-25T02:37:09.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-08-25T02:37:09.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: osdmap e92: 12 total, 12 up, 12 in 2023-08-25T02:37:09.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/3593818426' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-08-25T02:37:09.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mon[100457]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-08-25T02:37:09.982 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: ignoring --setuser ceph since I am not root 2023-08-25T02:37:09.982 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: ignoring --setgroup ceph since I am not root 2023-08-25T02:37:09.982 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-mds[126625]: -- 172.21.15.122:0/1983705303 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56371a8031e0 con 0x56371a7d8c00 2023-08-25T02:37:09.983 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:37:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: starting mds.i at 2023-08-25T02:37:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.515+0000 7f67df7fe700 1 -- 172.21.15.89:0/1920475775 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 104+0+0 (secure 0 0 0) 0x7f67e009a5f0 con 0x7f67e8105770 2023-08-25T02:37:10.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.518+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 msgr2=0x7f67d40a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:10.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.518+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 0x7f67d40a2580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009bc0 tx=0x7f67d8009260 comp rx=0 tx=0).stop 2023-08-25T02:37:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.518+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 msgr2=0x7f67e81a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.518+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f67e000b780 tx=0x7f67e00076f0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 shutdown_connections 2023-08-25T02:37:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f67d40a00c0 0x7f67d40a2580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e8105770 0x7f67e81a2760 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67e80ff760 0x7f67e81976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 --2- 172.21.15.89:0/1920475775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67e8100120 0x7f67e8197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 >> 172.21.15.89:0/1920475775 conn(0x7f67e80fb270 msgr2=0x7f67e80fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 shutdown_connections 2023-08-25T02:37:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.519+0000 7f67f07ce700 1 -- 172.21.15.89:0/1920475775 wait complete. 2023-08-25T02:37:10.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-08-25T02:37:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2025047158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 msgr2=0x7f8fdc10a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2025047158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc10a8b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009990 tx=0x7f8fcc009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2025047158 shutdown_connections 2023-08-25T02:37:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2025047158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc10ae80 0x7f8fdc10d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2025047158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc101ab0 0x7f8fdc101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2025047158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc10a8b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.661+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2025047158 >> 172.21.15.89:0/2025047158 conn(0x7f8fdc0fb270 msgr2=0x7f8fdc0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.662+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2025047158 shutdown_connections 2023-08-25T02:37:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.662+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2025047158 wait complete. 2023-08-25T02:37:10.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.662+0000 7f8fe3a9f700 1 Processor -- start 2023-08-25T02:37:10.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.662+0000 7f8fe3a9f700 1 -- start start 2023-08-25T02:37:10.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc101ab0 0x7f8fdc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:10.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc10ae80 0x7f8fdc1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc196f20 con 0x7f8fdc1023d0 2023-08-25T02:37:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc197090 con 0x7f8fdc101ab0 2023-08-25T02:37:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc197250 con 0x7f8fdc10ae80 2023-08-25T02:37:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49628/0 (socket says 172.21.15.89:49628) 2023-08-25T02:37:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 -- 172.21.15.89:0/2129206637 learned_addr learned my addr 172.21.15.89:0/2129206637 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe183b700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc101ab0 0x7f8fdc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe203c700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc10ae80 0x7f8fdc1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 -- 172.21.15.89:0/2129206637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc10ae80 msgr2=0x7f8fdc1968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc10ae80 0x7f8fdc1968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 -- 172.21.15.89:0/2129206637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc101ab0 msgr2=0x7f8fdc19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc101ab0 0x7f8fdc19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 -- 172.21.15.89:0/2129206637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0095f0 con 0x7f8fdc1023d0 2023-08-25T02:37:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe103a700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0095c0 tx=0x7f8fcc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc01d070 con 0x7f8fdc1023d0 2023-08-25T02:37:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc1974d0 con 0x7f8fdc1023d0 2023-08-25T02:37:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fcc005030 con 0x7f8fdc1023d0 2023-08-25T02:37:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.663+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc004a20 con 0x7f8fdc1023d0 2023-08-25T02:37:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.664+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc1a3830 con 0x7f8fdc1023d0 2023-08-25T02:37:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.664+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc105fa0 con 0x7f8fdc1023d0 2023-08-25T02:37:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.665+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f8fcc005410 con 0x7f8fdc1023d0 2023-08-25T02:37:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.666+0000 7f8fd2ffd700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 0x7f8fc80a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.666+0000 7f8fe183b700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 0x7f8fc80a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.666+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f8fcc0cc610 con 0x7f8fdc1023d0 2023-08-25T02:37:10.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.666+0000 7f8fe183b700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 0x7f8fc80a2580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8005950 tx=0x7f8fd800b410 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:10.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.670+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f8fcc0960b0 con 0x7f8fdc1023d0 2023-08-25T02:37:10.829 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:37:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-08-25T02:37:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-08-25T02:37:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: Cluster is now healthy 2023-08-25T02:37:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6835/2306424402,v1:172.21.15.121:6838/2306424402] up:boot 2023-08-25T02:37:10.830 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6838/744092453,v1:172.21.15.122:6839/744092453] up:boot 2023-08-25T02:37:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6835/3067502790,v1:172.21.15.122:6837/3067502790] up:boot 2023-08-25T02:37:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.122:6834/4242082110,v1:172.21.15.122:6836/4242082110] up:boot 2023-08-25T02:37:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6834/599678990,v1:172.21.15.121:6837/599678990] up:boot 2023-08-25T02:37:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: mds.? [v2:172.21.15.121:6836/3017513942,v1:172.21.15.121:6839/3017513942] up:boot 2023-08-25T02:37:10.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:37:10.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: fsmap 12 up:standby 2023-08-25T02:37:10.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:37:10.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:37:10.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:37:10.832 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:37:10.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:37:10.833 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:10 smithi089 ceph-mon[98732]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:37:10.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:10.859+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8fdc0676a0 con 0x7f8fdc1023d0 2023-08-25T02:37:10.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:37:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-08-25T02:37:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-08-25T02:37:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: Cluster is now healthy 2023-08-25T02:37:10.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6835/2306424402,v1:172.21.15.121:6838/2306424402] up:boot 2023-08-25T02:37:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6838/744092453,v1:172.21.15.122:6839/744092453] up:boot 2023-08-25T02:37:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6835/3067502790,v1:172.21.15.122:6837/3067502790] up:boot 2023-08-25T02:37:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.122:6834/4242082110,v1:172.21.15.122:6836/4242082110] up:boot 2023-08-25T02:37:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6834/599678990,v1:172.21.15.121:6837/599678990] up:boot 2023-08-25T02:37:10.970 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: mds.? [v2:172.21.15.121:6836/3017513942,v1:172.21.15.121:6839/3017513942] up:boot 2023-08-25T02:37:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:37:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: fsmap 12 up:standby 2023-08-25T02:37:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:37:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:37:10.971 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:37:10.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:37:10.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:37:10.972 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:10 smithi121 ceph-mon[100099]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:37:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-08-25T02:37:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-08-25T02:37:10.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-08-25T02:37:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: Cluster is now healthy 2023-08-25T02:37:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6835/2306424402,v1:172.21.15.121:6838/2306424402] up:boot 2023-08-25T02:37:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6838/744092453,v1:172.21.15.122:6839/744092453] up:boot 2023-08-25T02:37:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6835/3067502790,v1:172.21.15.122:6837/3067502790] up:boot 2023-08-25T02:37:10.979 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.122:6834/4242082110,v1:172.21.15.122:6836/4242082110] up:boot 2023-08-25T02:37:10.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6834/599678990,v1:172.21.15.121:6837/599678990] up:boot 2023-08-25T02:37:10.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: mds.? [v2:172.21.15.121:6836/3017513942,v1:172.21.15.121:6839/3017513942] up:boot 2023-08-25T02:37:10.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/1920475775' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-08-25T02:37:10.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: fsmap 12 up:standby 2023-08-25T02:37:10.980 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-08-25T02:37:10.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-08-25T02:37:10.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-08-25T02:37:10.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-08-25T02:37:10.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-08-25T02:37:10.981 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:10 smithi122 ceph-mon[100457]: from='mgr.24889 172.21.15.89:0/3025509756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-08-25T02:37:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.626+0000 7f8fd2ffd700 1 -- 172.21.15.89:0/2129206637 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8fcc0996d0 con 0x7f8fdc1023d0 2023-08-25T02:37:11.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 msgr2=0x7f8fc80a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:11.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 0x7f8fc80a2580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8005950 tx=0x7f8fd800b410 comp rx=0 tx=0).stop 2023-08-25T02:37:11.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 msgr2=0x7f8fdc19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:11.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0095c0 tx=0x7f8fcc00f690 comp rx=0 tx=0).stop 2023-08-25T02:37:11.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 shutdown_connections 2023-08-25T02:37:11.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f8fc80a00c0 0x7f8fc80a2580 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8005950 tx=0x7f8fd800b410 comp rx=0 tx=0).stop 2023-08-25T02:37:11.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fdc101ab0 0x7f8fdc19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fdc10ae80 0x7f8fdc1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 --2- 172.21.15.89:0/2129206637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fdc1023d0 0x7f8fdc19cd80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 >> 172.21.15.89:0/2129206637 conn(0x7f8fdc0fb270 msgr2=0x7f8fdc0fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 shutdown_connections 2023-08-25T02:37:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.631+0000 7f8fe3a9f700 1 -- 172.21.15.89:0/2129206637 wait complete. 2023-08-25T02:37:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-08-25T02:37:11.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-08-25T02:37:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 -- 172.21.15.89:0/3006318244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 msgr2=0x7f570c0ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:11.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 --2- 172.21.15.89:0/3006318244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0ffbe0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009990 tx=0x7f56fc009ca0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 -- 172.21.15.89:0/3006318244 shutdown_connections 2023-08-25T02:37:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 --2- 172.21.15.89:0/3006318244 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 0x7f570c10c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 --2- 172.21.15.89:0/3006318244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 0x7f570c105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 --2- 172.21.15.89:0/3006318244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0ffbe0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 -- 172.21.15.89:0/3006318244 >> 172.21.15.89:0/3006318244 conn(0x7f570c0fb270 msgr2=0x7f570c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 -- 172.21.15.89:0/3006318244 shutdown_connections 2023-08-25T02:37:11.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.783+0000 7f57111aa700 1 -- 172.21.15.89:0/3006318244 wait complete. 2023-08-25T02:37:11.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 Processor -- start 2023-08-25T02:37:11.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 -- start start 2023-08-25T02:37:11.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:11.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 0x7f570c0fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:11.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 0x7f570c1a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:11.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0fc680 con 0x7f570c0ff760 2023-08-25T02:37:11.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1a5030 con 0x7f570c100120 2023-08-25T02:37:11.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f57111aa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1a5170 con 0x7f570c105770 2023-08-25T02:37:11.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:11.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49640/0 (socket says 172.21.15.89:49640) 2023-08-25T02:37:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 -- 172.21.15.89:0/143383133 learned_addr learned my addr 172.21.15.89:0/143383133 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-08-25T02:37:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570b59e700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 0x7f570c1a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570a59c700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 0x7f570c0fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 -- 172.21.15.89:0/143383133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 msgr2=0x7f570c1a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 0x7f570c1a4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 -- 172.21.15.89:0/143383133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 msgr2=0x7f570c0fbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 0x7f570c0fbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.784+0000 7f570ad9d700 1 -- 172.21.15.89:0/143383133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0095f0 con 0x7f570c0ff760 2023-08-25T02:37:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f570ad9d700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f570000e9a0 tx=0x7f570000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700009790 con 0x7f570c0ff760 2023-08-25T02:37:11.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c1a53d0 con 0x7f570c0ff760 2023-08-25T02:37:11.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5700010430 con 0x7f570c0ff760 2023-08-25T02:37:11.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700018680 con 0x7f570c0ff760 2023-08-25T02:37:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.785+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c1a5900 con 0x7f570c0ff760 2023-08-25T02:37:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.786+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c0676a0 con 0x7f570c0ff760 2023-08-25T02:37:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.787+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 134802+0+0 (secure 0 0 0) 0x7f57000105a0 con 0x7f570c0ff760 2023-08-25T02:37:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.787+0000 7f56f3fff700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 0x7f56f40a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T02:37:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.788+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8246+0+0 (secure 0 0 0) 0x7f5700014070 con 0x7f570c0ff760 2023-08-25T02:37:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.788+0000 7f570a59c700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 0x7f56f40a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T02:37:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.788+0000 7f570a59c700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 0x7f56f40a2580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f570c0fd100 tx=0x7f56fc005fd0 comp rx=0 tx=0).ready entity=mgr.24889 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T02:37:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.791+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179682 (secure 0 0 0) 0x7f5700095410 con 0x7f570c0ff760 2023-08-25T02:37:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:11 smithi121 ceph-mon[100099]: pgmap v332: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:37:11.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:11 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:11 smithi122 ceph-mon[100457]: pgmap v332: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:37:11.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:11 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:11.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:11.986+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f570c10f910 con 0x7f570c0ff760 2023-08-25T02:37:12.079 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:11 smithi089 ceph-mon[98732]: pgmap v332: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-08-25T02:37:12.080 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:11 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:12.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.647+0000 7f56f3fff700 1 -- 172.21.15.89:0/143383133 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) v1 ==== 169+0+0 (secure 0 0 0) 0x7f57000260d0 con 0x7f570c0ff760 2023-08-25T02:37:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.651+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 msgr2=0x7f56f40a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:12.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.651+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 0x7f56f40a2580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f570c0fd100 tx=0x7f56fc005fd0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.651+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 msgr2=0x7f570c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T02:37:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.651+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f570000e9a0 tx=0x7f570000ecb0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 shutdown_connections 2023-08-25T02:37:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:6800/1258779165,v1:172.21.15.89:6801/1258779165] conn(0x7f56f40a00c0 0x7f56f40a2580 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f570c105770 0x7f570c1a4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570c0ff760 0x7f570c0fb910 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 --2- 172.21.15.89:0/143383133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f570c100120 0x7f570c0fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T02:37:12.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 >> 172.21.15.89:0/143383133 conn(0x7f570c0fb270 msgr2=0x7f570c067ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T02:37:12.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 shutdown_connections 2023-08-25T02:37:12.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-08-25T02:37:12.652+0000 7f57111aa700 1 -- 172.21.15.89:0/143383133 wait complete. 2023-08-25T02:37:12.656 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-08-25T02:37:12.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:37:12.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:37:12.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:37:12.802 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-08-25T02:37:12.803 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-08-25T02:37:12.820 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-08-25T02:37:12.836 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-08-25T02:37:12.869 INFO:tasks.cephadm:Stopping all daemons... 2023-08-25T02:37:12.870 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-08-25T02:37:12.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a 2023-08-25T02:37:12.894 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:12 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-08-25T02:37:12.894 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:12 smithi089 ceph-mon[98732]: osdmap e93: 12 total, 12 up, 12 in 2023-08-25T02:37:12.894 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:12 smithi089 ceph-mon[98732]: from='client.? 172.21.15.89:0/143383133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:12.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:12 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-08-25T02:37:12.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:12 smithi121 ceph-mon[100099]: osdmap e93: 12 total, 12 up, 12 in 2023-08-25T02:37:12.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:12 smithi121 ceph-mon[100099]: from='client.? 172.21.15.89:0/143383133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:12.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:12 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/2129206637' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-08-25T02:37:12.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:12 smithi122 ceph-mon[100457]: osdmap e93: 12 total, 12 up, 12 in 2023-08-25T02:37:12.978 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:12 smithi122 ceph-mon[100457]: from='client.? 172.21.15.89:0/143383133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-08-25T02:37:13.297 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:12 smithi089 systemd[1]: Stopping Ceph mon.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:13.579 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a[98709]: 2023-08-25T02:37:13.296+0000 7ff7d14c2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:13.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a[98709]: 2023-08-25T02:37:13.296+0000 7ff7d14c2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-08-25T02:37:13.580 INFO:journalctl@ceph.mon.a.smithi089.stdout:Aug 25 02:37:13 smithi089 bash[153614]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-a 2023-08-25T02:37:13.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.a.service' 2023-08-25T02:37:13.913 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:13.913 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-08-25T02:37:13.914 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-08-25T02:37:13.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.b 2023-08-25T02:37:14.219 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:13 smithi121 systemd[1]: Stopping Ceph mon.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:14.704 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-b[100074]: 2023-08-25T02:37:14.282+0000 7ff0b6360700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:14.705 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-b[100074]: 2023-08-25T02:37:14.282+0000 7ff0b6360700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-08-25T02:37:14.968 INFO:journalctl@ceph.mon.b.smithi121.stdout:Aug 25 02:37:14 smithi121 bash[131201]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-b 2023-08-25T02:37:15.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.b.service' 2023-08-25T02:37:15.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:15.090 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-08-25T02:37:15.090 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-08-25T02:37:15.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.c 2023-08-25T02:37:15.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:15 smithi122 systemd[1]: Stopping Ceph mon.c for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:15.753 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-c[100433]: 2023-08-25T02:37:15.487+0000 7f3803db0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:15.754 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-c[100433]: 2023-08-25T02:37:15.487+0000 7f3803db0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-08-25T02:37:16.313 INFO:journalctl@ceph.mon.c.smithi122.stdout:Aug 25 02:37:16 smithi122 bash[132765]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mon-c 2023-08-25T02:37:16.321 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mon.c.service' 2023-08-25T02:37:16.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:16.374 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-08-25T02:37:16.374 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-08-25T02:37:16.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x 2023-08-25T02:37:16.731 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:37:16 smithi089 systemd[1]: Stopping Ceph mgr.x for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:17.004 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:37:16 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x[98977]: 2023-08-25T02:37:16.729+0000 7fd26edb2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:17.260 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Aug 25 02:37:17 smithi089 bash[153745]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-x 2023-08-25T02:37:17.266 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.x.service' 2023-08-25T02:37:17.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:17.306 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-08-25T02:37:17.307 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-08-25T02:37:17.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.y 2023-08-25T02:37:17.718 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:37:17 smithi121 systemd[1]: Stopping Ceph mgr.y for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:18.012 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:37:17 smithi121 bash[131350]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-y 2023-08-25T02:37:18.012 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Aug 25 02:37:17 smithi121 systemd[1]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-08-25T02:37:18.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.y.service' 2023-08-25T02:37:18.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:18.233 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-08-25T02:37:18.233 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-08-25T02:37:18.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.z 2023-08-25T02:37:18.659 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:37:18 smithi122 systemd[1]: Stopping Ceph mgr.z for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:19.122 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:37:18 smithi122 bash[132910]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mgr-z 2023-08-25T02:37:19.123 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Aug 25 02:37:18 smithi122 systemd[1]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-08-25T02:37:19.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mgr.z.service' 2023-08-25T02:37:19.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:19.164 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-08-25T02:37:19.165 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-08-25T02:37:19.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.0 2023-08-25T02:37:19.514 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:37:19 smithi089 systemd[1]: Stopping Ceph osd.0 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:19.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:37:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:37:19.512+0000 7f65f6f72700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:19.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:37:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:37:19.512+0000 7f65f6f72700 -1 osd.0 94 *** Got signal Terminated *** 2023-08-25T02:37:19.830 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:37:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0[111831]: 2023-08-25T02:37:19.512+0000 7f65f6f72700 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:25.579 INFO:journalctl@ceph.osd.0.smithi089.stdout:Aug 25 02:37:25 smithi089 bash[153893]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-0 2023-08-25T02:37:26.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.0.service' 2023-08-25T02:37:26.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:26.604 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-08-25T02:37:26.604 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-08-25T02:37:26.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.1 2023-08-25T02:37:26.960 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:37:26 smithi121 systemd[1]: Stopping Ceph osd.1 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:27.218 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:37:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:37:26.958+0000 7f0191921700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:27.219 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:37:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:37:26.958+0000 7f0191921700 -1 osd.1 94 *** Got signal Terminated *** 2023-08-25T02:37:27.219 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:37:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1[105112]: 2023-08-25T02:37:26.958+0000 7f0191921700 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:32.719 INFO:journalctl@ceph.osd.1.smithi121.stdout:Aug 25 02:37:32 smithi121 bash[131491]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-1 2023-08-25T02:37:33.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.1.service' 2023-08-25T02:37:33.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:33.843 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-08-25T02:37:33.843 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-08-25T02:37:33.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.2 2023-08-25T02:37:34.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:37:33 smithi122 systemd[1]: Stopping Ceph osd.2 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:34.728 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:37:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:37:34.251+0000 7f48c49a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:34.728 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:37:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:37:34.251+0000 7f48c49a6700 -1 osd.2 94 *** Got signal Terminated *** 2023-08-25T02:37:34.729 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:37:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2[106232]: 2023-08-25T02:37:34.251+0000 7f48c49a6700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:40.228 INFO:journalctl@ceph.osd.2.smithi122.stdout:Aug 25 02:37:40 smithi122 bash[133057]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-2 2023-08-25T02:37:41.482 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.2.service' 2023-08-25T02:37:41.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:41.523 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-08-25T02:37:41.523 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-08-25T02:37:41.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.3 2023-08-25T02:37:41.829 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:37:41 smithi089 systemd[1]: Stopping Ceph osd.3 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:42.329 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:37:41 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:37:41.883+0000 7f047abc8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:42.329 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:37:41 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:37:41.883+0000 7f047abc8700 -1 osd.3 94 *** Got signal Terminated *** 2023-08-25T02:37:42.330 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:37:41 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3[117162]: 2023-08-25T02:37:41.883+0000 7f047abc8700 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:46.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:45.849+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:47.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:46 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:46.821+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:47.829 INFO:journalctl@ceph.osd.3.smithi089.stdout:Aug 25 02:37:47 smithi089 bash[154180]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-3 2023-08-25T02:37:47.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:47 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:47.543+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:47.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:47 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:47.536+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:48.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:47 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:47.817+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:48.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:48 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:48.022+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:48.719 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:48 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:48.496+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:48.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.3.service' 2023-08-25T02:37:48.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:48.768 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-08-25T02:37:48.768 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-08-25T02:37:48.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.4 2023-08-25T02:37:48.799 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:48 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:48.519+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:49.056 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:48 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:48.805+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:49.120 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:37:48 smithi121 systemd[1]: Stopping Ceph osd.4 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:49.468 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:37:49 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:37:49.119+0000 7f8872116700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:49.468 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:37:49 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:37:49.119+0000 7f8872116700 -1 osd.4 94 *** Got signal Terminated *** 2023-08-25T02:37:49.469 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:37:49 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4[110504]: 2023-08-25T02:37:49.119+0000 7f8872116700 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:49.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:49 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:49.055+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:49.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:49 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:49.537+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:49.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:49 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:49.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:50.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:49 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:49.826+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:50.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:50 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:50.050+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:50.545 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:50 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:50.120+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:50.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:50 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:50.544+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:50.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:50 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:50.476+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:51.074 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:50 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:50.777+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:51.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:51 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:51.073+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:51.544 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:51 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:51.155+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:51.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:51 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:51.460+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:51.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:51 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:51.543+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:52.115 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:51 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:51.793+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:52.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:52 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:52.114+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:52.528 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:52 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:52.113+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:52.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:52 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:52.450+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:52.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:52 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:52.528+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:53.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:52 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:52.827+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:53.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:53 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:53.079+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:53.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:53 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:53.079+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:53.573 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:53.102+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:53.574 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:53.102+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:53.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:53 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:53.427+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:53.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:53 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:53.427+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:53.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:53.573+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:53.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:53.573+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:54.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:53 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:53.873+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:54.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:54 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:54.053+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:54.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:54 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:54.053+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:54.542 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:54 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:54.079+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:54.543 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:54 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:54.079+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:54.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:54 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:54.446+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:54.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:54 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:54.446+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:54.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:54 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:54.301+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:54.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:54 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:54.541+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:54.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:54 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:54.541+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:55.218 INFO:journalctl@ceph.osd.4.smithi121.stdout:Aug 25 02:37:54 smithi121 bash[131761]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-4 2023-08-25T02:37:55.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:54 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:54.849+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:55.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:55 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:55.076+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:55.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:55 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:55.076+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:55.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:55 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:55.046+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:55.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:55 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:55.046+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:55.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:37:55 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:37:55.507+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:37:55.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:55 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:55.494+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:55.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:55 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:55.494+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:55.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:55 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:55.312+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:55.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:55 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:55.532+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:55.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:55 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:55.532+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:56.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.4.service' 2023-08-25T02:37:56.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:37:56.150 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-08-25T02:37:56.151 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-08-25T02:37:56.151 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.5 2023-08-25T02:37:56.182 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:55 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:55.872+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:56.182 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:55 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:55.872+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:37:56.183 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:56.119+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:56.183 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:56.119+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:56.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:56.049+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:56.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:56.049+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:56.478 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 systemd[1]: Stopping Ceph osd.5 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:37:56.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:56.263+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:56.828 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:56.534+0000 7fbb0c46e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:37:56.828 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:56.534+0000 7fbb0c46e700 -1 osd.5 94 *** Got signal Terminated *** 2023-08-25T02:37:56.828 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:56.534+0000 7fbb0c46e700 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:37:56.829 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:56.827+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:56.829 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:56 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:56.827+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:37:56.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:56.571+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:56.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:56 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:56.571+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:56.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:37:56 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:37:56.495+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:37:56.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:56 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:56.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:56.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:56 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:56.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:57.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:57 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:57.273+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:57.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:57 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:57.094+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:57.479 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:57 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:57.094+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:57.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:57.084+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:57.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:57.084+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:57.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:37:57 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:37:57.511+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:37:57.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:57 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:57.487+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:57.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:57 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:57.487+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:58.067 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:57.585+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:58.068 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:57 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:57.585+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:58.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:57 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:57.837+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:58.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:57 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:57.837+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:37:58.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:58.066+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:58.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:58.066+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:58.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:58 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:58.309+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:58.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:58 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:58.093+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:58.478 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:58 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:58.093+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:58.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:58.555+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:58.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:58.555+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:58.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:37:58 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:37:58.529+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:37:58.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:58 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:58.509+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:58.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:58 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:58.509+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:37:59.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:58 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:58.867+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:37:59.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:58 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:58.867+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:37:59.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:59 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:59.044+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:37:59.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:37:59 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:37:59.044+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:37:59.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:59.078+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:37:59.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:37:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:37:59.078+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:37:59.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:37:59 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:37:59.343+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:37:59.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:59.548+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:37:59.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:37:59 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:37:59.548+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:37:59.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:37:59 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:37:59.485+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:37:59.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:59 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:59.523+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:37:59.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:37:59 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:37:59.523+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:00.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:59 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:59.894+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:38:00.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:37:59 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:37:59.894+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:38:00.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:00.013+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:00.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:00.013+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:00.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:00.013+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:00.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:00.111+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:00.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:00.111+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:00.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:00.527+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:00.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:00.527+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:00.720 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:00.453+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:00.720 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:00 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:00.453+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:00.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:00.371+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:01.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:00.594+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:01.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:00.594+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:01.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:00 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:00.594+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:01.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:38:00.926+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:24.374935+0000 front 2023-08-25T02:37:24.374842+0000 (oldest deadline 2023-08-25T02:37:45.474836+0000) 2023-08-25T02:38:01.228 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:38:00 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5[111650]: 2023-08-25T02:38:00.926+0000 7fbb07261700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:28.976003+0000 front 2023-08-25T02:37:28.976175+0000 (oldest deadline 2023-08-25T02:37:54.876089+0000) 2023-08-25T02:38:01.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:01.021+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:01.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:01.021+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:01.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:01.021+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:01.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:01.134+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:01.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:01.135+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:01.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:01 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:01.339+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:01.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:01 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:01.484+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:01.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:01 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:01.484+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:01.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:01 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:01.499+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:01.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:01 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:01.499+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:02.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:01.612+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:02.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:01.612+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:02.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:01 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:01.612+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:02.478 INFO:journalctl@ceph.osd.5.smithi122.stdout:Aug 25 02:38:02 smithi122 bash[133330]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-5 2023-08-25T02:38:02.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:02.313+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:02.479 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:02.068+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:02.479 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:02.068+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:02.479 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:02 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:02.068+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:02.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:02.110+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:02.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:02.110+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:02.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:02.110+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:02.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:02 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:02.545+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:02.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:02 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:02.545+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:02.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:02 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:02.481+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:02.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:02 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:02.481+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:03.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:02.579+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:03.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:02.579+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:03.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:02 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:02.579+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:03.310 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:03.023+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:03.311 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:03.023+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:03.311 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:03.023+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:03.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:03.100+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:03.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:03.100+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:03.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:03.100+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:03.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:03.537+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:03.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:03.537+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:03.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:03.455+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:03.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:03.455+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:03.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:03 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:03.455+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:03.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:03 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:03.309+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:03.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.5.service' 2023-08-25T02:38:03.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:03.847 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-08-25T02:38:03.848 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-08-25T02:38:03.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.6 2023-08-25T02:38:03.878 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:03.592+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:03.878 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:03.592+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:03.879 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:03.592+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:04.198 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:03 smithi089 systemd[1]: Stopping Ceph osd.6 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:04.199 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.107+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:04.199 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.107+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:04.199 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.107+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:04.199 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.197+0000 7f8539081700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:04.199 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.197+0000 7f8539081700 -1 osd.6 94 *** Got signal Terminated *** 2023-08-25T02:38:04.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:04.034+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:04.310 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:04.034+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:04.310 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:04.034+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:04.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:04.197+0000 7f8539081700 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:04.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:04.308+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:04.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:04 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:04.308+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:04.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:04.512+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:04.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:04.512+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:04.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:04.482+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:04.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:04.482+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:04.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:04 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:04.482+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:05.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:04.596+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:05.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:04.596+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:05.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:04 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:04.596+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:05.354 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.001+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:05.355 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.001+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:05.355 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.001+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:05.558 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:05.086+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:05.559 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:05.086+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:05.559 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:05.086+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:05.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:05.354+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:05.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:05.354+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:05.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:05.558+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:05.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:05.558+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:05.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:05 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:05.558+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:05.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:05.558+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:05.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:05.558+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:05.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:05.533+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:05.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:05.533+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:05.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:05 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:05.533+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:06.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:06.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:06.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:05 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:05.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:06.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:06.076+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:06.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:06.076+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:06.330 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:06.076+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:06.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:06.403+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:06.812 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:06.403+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:06.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:06.575+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:06.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:06.575+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:06.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:06 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:06.575+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:06.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:06.524+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:06.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:06.524+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:06.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:06.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:06.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:06.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:06.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:06 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:06.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:07.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:06.933+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:07.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:06.933+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:07.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:06 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:06.933+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:07.561 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:07.106+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:07.561 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:07.106+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:07.562 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:07.106+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:07.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:07.412+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:07.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:07.412+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:07.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:07.412+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:07.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:07.560+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:07.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:07.560+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:07.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:07 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:07.560+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:07.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:07.507+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:07.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:07.507+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:07.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:07.509+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:07.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:07.509+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:07.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:07 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:07.509+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:08.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:07.904+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:08.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:07.904+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:08.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:07 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:07.904+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:08.579 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:08.134+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:08.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:08.134+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:08.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:08.134+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:08.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:08.461+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:08.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:08.461+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:08.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:08.540+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:08.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:08.540+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:08.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:08.540+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:08.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:08.453+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:08.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:08.453+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:08.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:08.453+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:09.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:08.608+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:09.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:08.608+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:09.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:08.608+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:09.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:08 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:08.608+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:09.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:08.875+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:09.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:08.875+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:09.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:08 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:08.875+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:09.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:09.175+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.565800+0000 front 2023-08-25T02:37:23.565854+0000 (oldest deadline 2023-08-25T02:37:49.466045+0000) 2023-08-25T02:38:09.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:09.175+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.967135+0000 front 2023-08-25T02:37:29.967211+0000 (oldest deadline 2023-08-25T02:37:52.867324+0000) 2023-08-25T02:38:09.580 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:09.175+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:37.969379+0000 front 2023-08-25T02:37:37.969423+0000 (oldest deadline 2023-08-25T02:38:01.469287+0000) 2023-08-25T02:38:09.581 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6[122531]: 2023-08-25T02:38:09.175+0000 7f8534675700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.970242+0000 front 2023-08-25T02:37:44.970271+0000 (oldest deadline 2023-08-25T02:38:09.070466+0000) 2023-08-25T02:38:09.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:09.450+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:09.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:09.450+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:09.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:09.553+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:09.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:09.553+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:09.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:09.553+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:09.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:09 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:09.553+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:09.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:09.446+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:09.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:09.446+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:09.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:09.446+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:09.971 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:09.626+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:09.971 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:09.626+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:09.971 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:09.626+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:09.972 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:09 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:09.626+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:10.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:09.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:10.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:09.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:10.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:09.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:10.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:09 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:09.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:10.329 INFO:journalctl@ceph.osd.6.smithi089.stdout:Aug 25 02:38:09 smithi089 bash[154446]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-6 2023-08-25T02:38:10.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:10.411+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:10.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:10.411+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:10.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:10.411+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:10.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:10.495+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:10.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:10.495+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:10.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:10.495+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:10.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:10.563+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:10.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:10.563+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:10.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:10.563+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:10.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:10.563+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:11.001 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:10 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:10.668+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:11.001 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:10 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:10.669+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:11.001 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:10 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:10.669+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:11.001 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:10 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:10.669+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:11.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:10.887+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:11.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:10.887+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:11.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:10.887+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:11.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:10 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:10.887+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:11.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.6.service' 2023-08-25T02:38:11.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:11.427 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-08-25T02:38:11.427 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-08-25T02:38:11.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.7 2023-08-25T02:38:11.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.459+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:11.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.459+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:11.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.459+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:11.720 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 systemd[1]: Stopping Ceph osd.7 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:11.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:11.522+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:11.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:11.522+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:11.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:11.522+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:11.721 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:11.522+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:11.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:11.394+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:11.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:11.394+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:11.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:11.394+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:12.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:11 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:11.696+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:12.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:11 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:11.696+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:12.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:11 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:11.696+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:12.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:11 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:11.696+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:12.218 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.773+0000 7f399acca700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:12.219 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.773+0000 7f399acca700 -1 osd.7 94 *** Got signal Terminated *** 2023-08-25T02:38:12.219 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:11 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:11.773+0000 7f399acca700 -1 osd.7 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:12.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:11.928+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:12.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:11.928+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:12.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:11.928+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:12.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:11 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:11.928+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:12.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:12.349+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:12.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:12.349+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:12.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:12.349+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:12.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:12.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:12.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:12.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:12.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:12.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:12.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:12.486+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:12.970 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:12.488+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:12.970 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:12.489+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:12.970 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:12 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:12.489+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:13.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:12 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:12.673+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:13.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:12 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:12.673+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:13.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:12 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:12.673+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:13.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:12 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:12.673+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:13.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:12.969+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:13.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:12.969+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:13.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:12.969+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:13.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:12 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:12.969+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:13.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:13.307+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:13.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:13.307+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:13.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:13.307+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:13.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:13.508+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:13.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:13.508+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:13.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:13.508+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:13.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:13.508+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:13.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:13.508+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:13.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:13.508+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:13.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:13 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:13.508+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:14.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:13.694+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:14.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:13.694+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:14.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:13.694+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:14.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:13 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:13.694+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:14.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:13.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:14.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:13.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:14.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:13.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:14.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:13 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:13.955+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:14.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:14.297+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:14.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:14.297+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:14.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:14.298+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:14.968 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:14.557+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:14.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:14.557+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:14.969 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:14.557+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:14.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:14.477+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:14.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:14.477+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:14.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:14.477+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:14.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:14.477+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:15.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:14 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:14.720+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:15.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:14 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:14.720+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:15.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:14 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:14.720+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:15.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:14 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:14.720+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:15.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:14.952+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:15.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:14.952+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:15.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:14.952+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:15.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:14 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:14.952+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:15.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:15.443+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:15.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:15.443+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:15.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:15.443+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:15.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:15.443+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:15.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:15.443+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:15.720 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:15.560+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:15.720 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:15.560+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:15.721 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:15 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:15.560+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:15.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:15.266+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:15.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:15.266+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:15.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:15.266+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:16.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:15 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:15.698+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:16.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:15 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:15.699+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:16.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:15 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:15.699+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:16.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:15 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:15.699+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:16.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:15.930+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:16.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:15.930+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:16.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:15.930+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:16.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:15 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:15.930+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:16.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:16.548+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:31.359315+0000 front 2023-08-25T02:37:31.359342+0000 (oldest deadline 2023-08-25T02:37:55.459517+0000) 2023-08-25T02:38:16.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:16.548+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:35.460363+0000 front 2023-08-25T02:37:35.460585+0000 (oldest deadline 2023-08-25T02:38:00.160389+0000) 2023-08-25T02:38:16.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:16.548+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.661604+0000 front 2023-08-25T02:37:43.661568+0000 (oldest deadline 2023-08-25T02:38:09.561470+0000) 2023-08-25T02:38:16.719 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7[116052]: 2023-08-25T02:38:16.548+0000 7f3996abf700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.462592+0000 front 2023-08-25T02:37:52.462626+0000 (oldest deadline 2023-08-25T02:38:15.863278+0000) 2023-08-25T02:38:16.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:16.448+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:16.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:16.448+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:16.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:16.448+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:16.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:16.448+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:16.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:16 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:16.448+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:16.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:16.264+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:16.730 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:16.264+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:16.730 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:16.264+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:17.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:16 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:16.747+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:17.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:16 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:16.747+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:17.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:16 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:16.747+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:17.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:16 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:16.747+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:17.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:16.902+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:17.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:16.902+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:17.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:16.902+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:17.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:16 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:16.902+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:17.718 INFO:journalctl@ceph.osd.7.smithi121.stdout:Aug 25 02:38:17 smithi121 bash[132034]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-7 2023-08-25T02:38:17.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:17 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:17.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:17.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:17 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:17.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:17.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:17 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:17.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:17.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:17 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:17.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:17.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:17 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:17.479+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:17.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:17.237+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:17.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:17.237+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:17.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:17.237+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:18.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:17 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:17.729+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:18.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:17 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:17.729+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:18.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:17 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:17.729+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:18.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:17 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:17.729+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:18.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:17.861+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:18.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:17.861+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:18.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:17.861+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:18.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:17.861+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:18.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:17 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:17.861+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:18.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:18.516+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:18.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:18.516+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:18.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:18.516+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:18.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:18.516+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:18.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:18.516+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:18.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.7.service' 2023-08-25T02:38:18.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:18.255+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:18.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:18.255+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:18.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:18.255+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:18.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:18.769 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-08-25T02:38:18.769 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-08-25T02:38:18.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.8 2023-08-25T02:38:19.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:18.740+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:19.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:18.740+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:19.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:18.740+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:19.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:18 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:18.740+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:19.130 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:18 smithi122 systemd[1]: Stopping Ceph osd.8 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:19.131 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.129+0000 7feb3aeed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:19.131 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:18.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:19.131 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:18.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:19.132 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:18.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:19.132 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:18.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:19.132 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:18 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:18.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:19.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.129+0000 7feb3aeed700 -1 osd.8 94 *** Got signal Terminated *** 2023-08-25T02:38:19.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.129+0000 7feb3aeed700 -1 osd.8 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:19.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.262+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:19.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.262+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:19.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.262+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:19.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:19.262+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:53.173130+0000 front 2023-08-25T02:37:53.173249+0000 (oldest deadline 2023-08-25T02:38:19.073077+0000) 2023-08-25T02:38:19.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:19 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:19.470+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:19.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:19 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:19.470+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:19.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:19 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:19.470+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:19.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:19 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:19.470+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:19.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:19 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:19.470+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:20.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:19.741+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:20.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:19.741+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:20.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:19.741+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:20.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:19 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:19.741+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:20.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:19.796+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:20.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:19.796+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:20.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:19.796+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:20.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:19.796+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:20.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:19 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:19.796+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:20.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:20 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:20.451+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:20.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:20 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:20.451+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:20.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:20 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:20.451+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:20.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:20 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:20.451+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:20.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:20 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:20.451+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:20.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:20.231+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:20.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:20.231+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:20.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:20.231+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:20.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:20.231+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:53.173130+0000 front 2023-08-25T02:37:53.173249+0000 (oldest deadline 2023-08-25T02:38:19.073077+0000) 2023-08-25T02:38:21.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:20 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:20.697+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:21.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:20 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:20.697+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:21.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:20 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:20.697+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:21.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:20 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:20.697+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:21.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:20.803+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:21.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:20.803+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:21.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:20.803+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:21.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:20.803+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:21.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:20 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:20.803+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:21.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:21.272+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:21.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:21.272+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:21.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:21.272+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:21.730 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:21.272+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:53.173130+0000 front 2023-08-25T02:37:53.173249+0000 (oldest deadline 2023-08-25T02:38:19.073077+0000) 2023-08-25T02:38:21.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:21 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:21.474+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:21.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:21 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:21.474+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:21.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:21 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:21.474+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:21.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:21 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:21.474+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:21.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:21 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:21.474+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:22.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:21.661+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:22.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:21.661+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:22.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:21.661+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:22.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:21 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:21.661+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:22.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:21.816+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:22.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:21.816+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:22.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:21.816+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:22.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:21.816+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:22.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:21 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:21.816+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:22.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:22 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:22.425+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:22.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:22 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:22.425+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:22.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:22 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:22.425+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:22.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:22 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:22.425+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:22.721 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:22 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:22.425+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:22.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:22.260+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:22.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:22.260+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:22.728 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:22.260+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:22.729 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:22.260+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:53.173130+0000 front 2023-08-25T02:37:53.173249+0000 (oldest deadline 2023-08-25T02:38:19.073077+0000) 2023-08-25T02:38:23.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:22.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:23.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:22.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:23.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:22.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:23.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:22 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:22.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:23.216 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:22.814+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:23.217 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:22.814+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:23.217 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:22.814+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:23.217 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:22.814+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:23.218 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:22.814+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:23.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:23.215+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:29.968347+0000 front 2023-08-25T02:37:29.968429+0000 (oldest deadline 2023-08-25T02:37:54.068338+0000) 2023-08-25T02:38:23.478 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:23.215+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:38.070133+0000 front 2023-08-25T02:37:38.070109+0000 (oldest deadline 2023-08-25T02:38:03.370332+0000) 2023-08-25T02:38:23.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:23.215+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:43.871626+0000 front 2023-08-25T02:37:43.871717+0000 (oldest deadline 2023-08-25T02:38:07.371591+0000) 2023-08-25T02:38:23.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:23.215+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:53.173130+0000 front 2023-08-25T02:37:53.173249+0000 (oldest deadline 2023-08-25T02:38:19.073077+0000) 2023-08-25T02:38:23.479 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8[117203]: 2023-08-25T02:38:23.215+0000 7feb37ce4700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:59.073403+0000 front 2023-08-25T02:37:59.073431+0000 (oldest deadline 2023-08-25T02:38:22.573451+0000) 2023-08-25T02:38:23.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:23.408+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:23.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:23.408+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:23.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:23.408+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:23.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:23.408+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:23.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:23 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:23.408+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:24.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:23.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:24.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:23.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:24.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:23.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:24.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:23 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:23.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:24.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:24.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:24.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:24.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:24.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:24.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:23 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:23.829+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:24.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:24.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:24.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:24.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:24.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:24.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:24 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:24.433+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:24.908 INFO:journalctl@ceph.osd.8.smithi122.stdout:Aug 25 02:38:24 smithi122 bash[133602]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-8 2023-08-25T02:38:24.909 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:24.909 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:24.909 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:24.909 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:24.910 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:24.910 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:24 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:24.866+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:25.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:24.653+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:25.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:24.653+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:25.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:24.653+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:25.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:24 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:24.653+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:25.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:25.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:25.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:25.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:25.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:25.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:25 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:25.421+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:26.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:25.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:26.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:25.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:26.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:25.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:26.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:25 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:25.695+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:26.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:26.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:26.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:26.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:26.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:26.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:25 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:25.890+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:26.469 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.8.service' 2023-08-25T02:38:26.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:26.511 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-08-25T02:38:26.512 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-08-25T02:38:26.513 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.9 2023-08-25T02:38:26.718 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:26.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:26.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:26.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:26.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:26.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:26 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:26.416+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:26.829 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 systemd[1]: Stopping Ceph osd.9 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:26.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.711+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:26.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.711+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:26.830 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.711+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:26.831 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.711+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:27.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:27.243 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:27.244 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:27.244 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:27.244 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:27.245 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:26.900+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:27.329 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.868+0000 7f79b551f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:27.330 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.868+0000 7f79b551f700 -1 osd.9 94 *** Got signal Terminated *** 2023-08-25T02:38:27.330 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:26 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:26.868+0000 7f79b551f700 -1 osd.9 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:27.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:27.719 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:27.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:27.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:27.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:27.720 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:27 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:27.467+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:28.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:27.701+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:28.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:27.701+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:28.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:27.701+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:28.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:27.701+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:28.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:27 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:27.701+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.178469+0000 front 2023-08-25T02:38:01.178383+0000 (oldest deadline 2023-08-25T02:38:26.979148+0000) 2023-08-25T02:38:28.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:28.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:28.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:28.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:28.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:28.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:27 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:27.925+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:28.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:28.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:28.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:28.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:28.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:28.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:28 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:28.503+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:29.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:28.689+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:29.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:28.689+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:29.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:28.689+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:29.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:28.689+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:29.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:28 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:28.689+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.178469+0000 front 2023-08-25T02:38:01.178383+0000 (oldest deadline 2023-08-25T02:38:26.979148+0000) 2023-08-25T02:38:29.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:29.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:29.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:29.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:29.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:29.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:28 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:28.926+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:29.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:29.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:29.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:29.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:29.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:29.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:29 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:29.514+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:30.079 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:29.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:30.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:29.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:30.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:29.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:30.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:29.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:30.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:29 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:29.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.178469+0000 front 2023-08-25T02:38:01.178383+0000 (oldest deadline 2023-08-25T02:38:26.979148+0000) 2023-08-25T02:38:30.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:30.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:30.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:30.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:30.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:30.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:29 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:29.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:30.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:30.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:30.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:30.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:30.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:30.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:30.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:30 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:30.518+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:31.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:31.082 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:31.082 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:31.082 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:31.083 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.178469+0000 front 2023-08-25T02:38:01.178383+0000 (oldest deadline 2023-08-25T02:38:26.979148+0000) 2023-08-25T02:38:31.083 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:30 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:30.665+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:06.979662+0000 front 2023-08-25T02:38:06.979687+0000 (oldest deadline 2023-08-25T02:38:30.479770+0000) 2023-08-25T02:38:31.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:31.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:31.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:31.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:31.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:31.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:30.922+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:31.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:31.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:31.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:31.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:31.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:31.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:31.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:31 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:31.512+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:32.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.371904+0000 front 2023-08-25T02:37:23.371935+0000 (oldest deadline 2023-08-25T02:37:47.472237+0000) 2023-08-25T02:38:32.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.472912+0000 front 2023-08-25T02:37:27.472804+0000 (oldest deadline 2023-08-25T02:37:52.772770+0000) 2023-08-25T02:38:32.080 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.274054+0000 front 2023-08-25T02:37:36.274071+0000 (oldest deadline 2023-08-25T02:38:00.373886+0000) 2023-08-25T02:38:32.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.875744+0000 front 2023-08-25T02:37:44.875768+0000 (oldest deadline 2023-08-25T02:38:08.375816+0000) 2023-08-25T02:38:32.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.178469+0000 front 2023-08-25T02:38:01.178383+0000 (oldest deadline 2023-08-25T02:38:26.979148+0000) 2023-08-25T02:38:32.081 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:31 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9[127997]: 2023-08-25T02:38:31.713+0000 7f79b0312700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:06.979662+0000 front 2023-08-25T02:38:06.979687+0000 (oldest deadline 2023-08-25T02:38:30.479770+0000) 2023-08-25T02:38:32.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:32.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:32.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:32.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:32.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:32.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:31 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:31.932+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:32.551 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:32 smithi089 bash[154706]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-9 2023-08-25T02:38:32.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:32.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:32.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:32.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:32.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:32.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:32.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:32 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:32.537+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:33.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:33.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:33.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:33.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:33.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:33.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:32 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:32.912+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:33.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.9.service' 2023-08-25T02:38:33.738 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:33 smithi089 systemd[1]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.9.service: Succeeded. 2023-08-25T02:38:33.739 INFO:journalctl@ceph.osd.9.smithi089.stdout:Aug 25 02:38:33 smithi089 systemd[1]: Stopped Ceph osd.9 for 7eb69934-42ed-11ee-9b3d-001a4aab830c. 2023-08-25T02:38:33.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:33.753 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-08-25T02:38:33.753 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-08-25T02:38:33.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.10 2023-08-25T02:38:33.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:33.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:33.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:33.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:33.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:33.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:33.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:33.585+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:33.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:33 smithi121 systemd[1]: Stopping Ceph osd.10 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:34.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:34.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:34.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:34.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:34.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:34.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:33 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:33.908+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:34.468 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.102+0000 7f755a0e8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:34.469 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.102+0000 7f755a0e8700 -1 osd.10 94 *** Got signal Terminated *** 2023-08-25T02:38:34.469 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.102+0000 7f755a0e8700 -1 osd.10 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:34.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:34.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:34.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:34.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:34.972 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:34.972 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:34.972 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:34 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:34.568+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:35.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:35.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:35.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:35.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:35.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:35.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:34.889+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:35.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:35.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:35.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:35.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:35.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:35.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:35.971 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:35 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:35.608+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:36.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:36.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:36.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:36.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:36.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:36.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:35 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:35.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:36.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:36.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:36.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:36.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:36.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:36.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:36.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:36 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:36.654+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:37.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:37.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:37.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:37.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:37.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:37.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:36 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:36.843+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:37.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:37.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:37.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:37.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:37.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:37.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:37.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:37 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:37.686+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:38.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:38.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:38.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:38.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:38.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:38.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:37 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:37.832+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:38.968 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:21.528194+0000 front 2023-08-25T02:37:21.528072+0000 (oldest deadline 2023-08-25T02:37:46.828105+0000) 2023-08-25T02:38:38.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:26.828487+0000 front 2023-08-25T02:37:26.828476+0000 (oldest deadline 2023-08-25T02:37:53.229268+0000) 2023-08-25T02:38:38.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.630877+0000 front 2023-08-25T02:37:36.630844+0000 (oldest deadline 2023-08-25T02:38:02.530706+0000) 2023-08-25T02:38:38.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:45.932316+0000 front 2023-08-25T02:37:45.932248+0000 (oldest deadline 2023-08-25T02:38:09.432161+0000) 2023-08-25T02:38:38.969 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.833440+0000 front 2023-08-25T02:37:52.833509+0000 (oldest deadline 2023-08-25T02:38:15.133611+0000) 2023-08-25T02:38:38.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:38:01.435859+0000 front 2023-08-25T02:38:01.435764+0000 (oldest deadline 2023-08-25T02:38:24.335582+0000) 2023-08-25T02:38:38.970 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:38 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10[121499]: 2023-08-25T02:38:38.645+0000 7f7554edb700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2023-08-25T02:38:09.036660+0000 front 2023-08-25T02:38:09.036615+0000 (oldest deadline 2023-08-25T02:38:30.136574+0000) 2023-08-25T02:38:39.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:39.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:39.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:39.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:39.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:39.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:38 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:38.824+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:40.218 INFO:journalctl@ceph.osd.10.smithi121.stdout:Aug 25 02:38:39 smithi121 bash[132295]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-10 2023-08-25T02:38:40.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:40.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:40.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:40.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:40.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:40.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:39 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:39.784+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:41.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.10.service' 2023-08-25T02:38:41.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:41.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:41.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:41.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:41.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:41.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:40 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:40.808+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:41.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:41.233 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-08-25T02:38:41.234 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-08-25T02:38:41.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.11 2023-08-25T02:38:41.595 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 systemd[1]: Stopping Ceph osd.11 for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:41.847 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.593+0000 7f68baac9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:41.847 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.593+0000 7f68baac9700 -1 osd.11 94 *** Got signal Terminated *** 2023-08-25T02:38:41.848 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.593+0000 7f68baac9700 -1 osd.11 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T02:38:42.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:42.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:42.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:42.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:42.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:42.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:41 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:41.846+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:43.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:43.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:43.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:43.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:43.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:43.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:43.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:42 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:42.881+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6820 osd.7 since back 2023-08-25T02:38:16.591448+0000 front 2023-08-25T02:38:16.591428+0000 (oldest deadline 2023-08-25T02:38:42.491251+0000) 2023-08-25T02:38:44.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:44.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:44.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:44.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:44.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:44.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:44.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:43 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:43.871+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6820 osd.7 since back 2023-08-25T02:38:16.591448+0000 front 2023-08-25T02:38:16.591428+0000 (oldest deadline 2023-08-25T02:38:42.491251+0000) 2023-08-25T02:38:45.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:45.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:45.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:45.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:45.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:45.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:45.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:44 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:44.892+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6820 osd.7 since back 2023-08-25T02:38:16.591448+0000 front 2023-08-25T02:38:16.591428+0000 (oldest deadline 2023-08-25T02:38:42.491251+0000) 2023-08-25T02:38:46.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2023-08-25T02:37:23.582928+0000 front 2023-08-25T02:37:23.583047+0000 (oldest deadline 2023-08-25T02:37:47.682895+0000) 2023-08-25T02:38:46.228 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2023-08-25T02:37:27.683692+0000 front 2023-08-25T02:37:27.683471+0000 (oldest deadline 2023-08-25T02:37:52.983448+0000) 2023-08-25T02:38:46.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2023-08-25T02:37:36.484526+0000 front 2023-08-25T02:37:36.484566+0000 (oldest deadline 2023-08-25T02:37:59.984746+0000) 2023-08-25T02:38:46.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2023-08-25T02:37:44.685685+0000 front 2023-08-25T02:37:44.685589+0000 (oldest deadline 2023-08-25T02:38:09.385700+0000) 2023-08-25T02:38:46.229 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2023-08-25T02:37:52.287126+0000 front 2023-08-25T02:37:52.287041+0000 (oldest deadline 2023-08-25T02:38:16.986996+0000) 2023-08-25T02:38:46.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-08-25T02:37:56.987345+0000 front 2023-08-25T02:37:56.987391+0000 (oldest deadline 2023-08-25T02:38:22.887526+0000) 2023-08-25T02:38:46.230 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:45 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11[122757]: 2023-08-25T02:38:45.907+0000 7f68b70bf700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.121:6820 osd.7 since back 2023-08-25T02:38:16.591448+0000 front 2023-08-25T02:38:16.591428+0000 (oldest deadline 2023-08-25T02:38:42.491251+0000) 2023-08-25T02:38:47.728 INFO:journalctl@ceph.osd.11.smithi122.stdout:Aug 25 02:38:47 smithi122 bash[133861]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-osd-11 2023-08-25T02:38:48.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@osd.11.service' 2023-08-25T02:38:48.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:48.940 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-08-25T02:38:48.941 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-08-25T02:38:48.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.a 2023-08-25T02:38:49.282 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:38:48 smithi089 systemd[1]: Stopping Ceph mds.a for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:49.579 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:38:49 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-a[133372]: 2023-08-25T02:38:49.281+0000 7fb2cb402700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:49.580 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:38:49 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-a[133372]: 2023-08-25T02:38:49.281+0000 7fb2cb402700 -1 mds.a *** got signal Terminated *** 2023-08-25T02:38:53.544 INFO:journalctl@ceph.mds.a.smithi089.stdout:Aug 25 02:38:53 smithi089 bash[154978]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-a 2023-08-25T02:38:53.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.a.service' 2023-08-25T02:38:53.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:53.588 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-08-25T02:38:53.588 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-08-25T02:38:53.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.d 2023-08-25T02:38:53.829 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:38:53 smithi089 systemd[1]: Stopping Ceph mds.d for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:54.329 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:38:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-d[132373]: 2023-08-25T02:38:53.980+0000 7f42dd710700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:54.329 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:38:53 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-d[132373]: 2023-08-25T02:38:53.980+0000 7f42dd710700 -1 mds.d *** got signal Terminated *** 2023-08-25T02:38:57.829 INFO:journalctl@ceph.mds.d.smithi089.stdout:Aug 25 02:38:57 smithi089 bash[155117]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-d 2023-08-25T02:38:57.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.d.service' 2023-08-25T02:38:57.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:38:57.921 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-08-25T02:38:57.922 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-08-25T02:38:57.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.g 2023-08-25T02:38:58.306 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:38:57 smithi089 systemd[1]: Stopping Ceph mds.g for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:38:58.579 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:38:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-g[132986]: 2023-08-25T02:38:58.304+0000 7f6f76c9f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:38:58.580 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:38:58 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-g[132986]: 2023-08-25T02:38:58.304+0000 7f6f76c9f700 -1 mds.g *** got signal Terminated *** 2023-08-25T02:39:02.773 INFO:journalctl@ceph.mds.g.smithi089.stdout:Aug 25 02:39:02 smithi089 bash[155269]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-g 2023-08-25T02:39:02.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.g.service' 2023-08-25T02:39:02.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:02.813 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-08-25T02:39:02.813 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-08-25T02:39:02.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.j 2023-08-25T02:39:03.079 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:39:02 smithi089 systemd[1]: Stopping Ceph mds.j for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:03.579 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:39:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-j[132013]: 2023-08-25T02:39:03.179+0000 7fb4d9dd1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:03.580 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:39:03 smithi089 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-j[132013]: 2023-08-25T02:39:03.179+0000 7fb4d9dd1700 -1 mds.j *** got signal Terminated *** 2023-08-25T02:39:07.695 INFO:journalctl@ceph.mds.j.smithi089.stdout:Aug 25 02:39:07 smithi089 bash[155416]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-j 2023-08-25T02:39:07.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.j.service' 2023-08-25T02:39:07.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:07.734 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-08-25T02:39:07.735 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-08-25T02:39:07.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.b 2023-08-25T02:39:08.138 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:39:07 smithi121 systemd[1]: Stopping Ceph mds.b for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:08.468 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:39:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: 2023-08-25T02:39:08.136+0000 7f8410d70700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:08.469 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:39:08 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b[125271]: 2023-08-25T02:39:08.136+0000 7f8410d70700 -1 mds.b *** got signal Terminated *** 2023-08-25T02:39:10.215 INFO:journalctl@ceph.mds.b.smithi121.stdout:Aug 25 02:39:09 smithi121 bash[132567]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-b 2023-08-25T02:39:10.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.b.service' 2023-08-25T02:39:10.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:10.260 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-08-25T02:39:10.260 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-08-25T02:39:10.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.e 2023-08-25T02:39:10.468 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:39:10 smithi121 systemd[1]: Stopping Ceph mds.e for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:10.968 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:39:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: 2023-08-25T02:39:10.635+0000 7f6cf994d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:10.968 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:39:10 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e[124591]: 2023-08-25T02:39:10.635+0000 7f6cf994d700 -1 mds.e *** got signal Terminated *** 2023-08-25T02:39:14.185 INFO:journalctl@ceph.mds.e.smithi121.stdout:Aug 25 02:39:13 smithi121 bash[132718]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-e 2023-08-25T02:39:14.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.e.service' 2023-08-25T02:39:14.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:14.226 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-08-25T02:39:14.226 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-08-25T02:39:14.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.h 2023-08-25T02:39:14.468 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:39:14 smithi121 systemd[1]: Stopping Ceph mds.h for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:14.968 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:39:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: 2023-08-25T02:39:14.625+0000 7f59c1792700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:14.969 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:39:14 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h[124932]: 2023-08-25T02:39:14.625+0000 7f59c1792700 -1 mds.h *** got signal Terminated *** 2023-08-25T02:39:18.218 INFO:journalctl@ceph.mds.h.smithi121.stdout:Aug 25 02:39:17 smithi121 bash[132855]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-h 2023-08-25T02:39:18.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.h.service' 2023-08-25T02:39:18.329 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:18.329 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-08-25T02:39:18.330 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-08-25T02:39:18.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.k 2023-08-25T02:39:18.718 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:39:18 smithi121 systemd[1]: Stopping Ceph mds.k for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:19.218 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:39:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-k[124250]: 2023-08-25T02:39:18.725+0000 7fa180b80700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:19.219 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:39:18 smithi121 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-k[124250]: 2023-08-25T02:39:18.725+0000 7fa180b80700 -1 mds.k *** got signal Terminated *** 2023-08-25T02:39:21.569 INFO:journalctl@ceph.mds.k.smithi121.stdout:Aug 25 02:39:21 smithi121 bash[133010]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-k 2023-08-25T02:39:21.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.k.service' 2023-08-25T02:39:21.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:21.801 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-08-25T02:39:21.801 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-08-25T02:39:21.801 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.c 2023-08-25T02:39:22.165 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:39:21 smithi122 systemd[1]: Stopping Ceph mds.c for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:22.478 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:39:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: 2023-08-25T02:39:22.163+0000 7f89c14a5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:22.478 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:39:22 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c[126262]: 2023-08-25T02:39:22.163+0000 7f89c14a5700 -1 mds.c *** got signal Terminated *** 2023-08-25T02:39:25.978 INFO:journalctl@ceph.mds.c.smithi122.stdout:Aug 25 02:39:25 smithi122 bash[134141]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-c 2023-08-25T02:39:26.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.c.service' 2023-08-25T02:39:26.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:26.160 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-08-25T02:39:26.160 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-08-25T02:39:26.161 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.f 2023-08-25T02:39:26.478 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:39:26 smithi122 systemd[1]: Stopping Ceph mds.f for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:26.978 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:39:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: 2023-08-25T02:39:26.544+0000 7f399e0aa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:26.978 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:39:26 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f[125922]: 2023-08-25T02:39:26.544+0000 7f399e0aa700 -1 mds.f *** got signal Terminated *** 2023-08-25T02:39:29.978 INFO:journalctl@ceph.mds.f.smithi122.stdout:Aug 25 02:39:29 smithi122 bash[134281]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-f 2023-08-25T02:39:30.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.f.service' 2023-08-25T02:39:30.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:30.132 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-08-25T02:39:30.133 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-08-25T02:39:30.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.i 2023-08-25T02:39:30.478 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:39:30 smithi122 systemd[1]: Stopping Ceph mds.i for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:30.978 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:39:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: 2023-08-25T02:39:30.520+0000 7fb820302700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:30.979 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:39:30 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i[126601]: 2023-08-25T02:39:30.520+0000 7fb820302700 -1 mds.i *** got signal Terminated *** 2023-08-25T02:39:34.209 INFO:journalctl@ceph.mds.i.smithi122.stdout:Aug 25 02:39:33 smithi122 bash[134419]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-i 2023-08-25T02:39:34.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.i.service' 2023-08-25T02:39:34.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:34.248 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-08-25T02:39:34.248 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-08-25T02:39:34.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.l 2023-08-25T02:39:34.478 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:39:34 smithi122 systemd[1]: Stopping Ceph mds.l for 7eb69934-42ed-11ee-9b3d-001a4aab830c... 2023-08-25T02:39:34.978 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:39:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-l[125579]: 2023-08-25T02:39:34.635+0000 7f37cfc9e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-08-25T02:39:34.978 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:39:34 smithi122 ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-l[125579]: 2023-08-25T02:39:34.635+0000 7f37cfc9e700 -1 mds.l *** got signal Terminated *** 2023-08-25T02:39:38.978 INFO:journalctl@ceph.mds.l.smithi122.stdout:Aug 25 02:39:38 smithi122 bash[134565]: ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c-mds-l 2023-08-25T02:39:39.207 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eb69934-42ed-11ee-9b3d-001a4aab830c@mds.l.service' 2023-08-25T02:39:39.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-08-25T02:39:39.240 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-08-25T02:39:39.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force --keep-logs 2023-08-25T02:39:39.541 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:39:41.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force --keep-logs 2023-08-25T02:39:41.405 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:39:42.979 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force --keep-logs 2023-08-25T02:39:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:39:44.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:39:44.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:39:45.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-08-25T02:39:45.046 INFO:tasks.cephadm:Archiving crash dumps... 2023-08-25T02:39:45.049 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi089/crash 2023-08-25T02:39:45.050 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash -- . 2023-08-25T02:39:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:tar: /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash: Cannot open: No such file or directory 2023-08-25T02:39:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:tar: Error is not recoverable: exiting now 2023-08-25T02:39:45.131 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi121/crash 2023-08-25T02:39:45.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash -- . 2023-08-25T02:39:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash: Cannot open: No such file or directory 2023-08-25T02:39:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2023-08-25T02:39:45.260 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi122/crash 2023-08-25T02:39:45.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash -- . 2023-08-25T02:39:45.315 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/crash: Cannot open: No such file or directory 2023-08-25T02:39:45.316 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2023-08-25T02:39:45.317 INFO:tasks.cephadm:Checking cluster log for badness... 2023-08-25T02:39:45.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-08-25T02:39:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:grep: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph.log: No such file or directory 2023-08-25T02:39:45.354 INFO:tasks.cephadm:Compressing logs... 2023-08-25T02:39:45.354 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-25T02:39:45.396 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-25T02:39:45.399 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-25T02:39:45.424 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-08-25T02:39:45.425 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2023-08-25T02:39:45.425 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.0.log 2023-08-25T02:39:45.426 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.g.log 2023-08-25T02:39:45.426 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.a.log 2023-08-25T02:39:45.426 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.6.log 2023-08-25T02:39:45.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.3.log 2023-08-25T02:39:45.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.d.log 2023-08-25T02:39:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-08-25T02:39:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2023-08-25T02:39:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.10.log 2023-08-25T02:39:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.h.log 2023-08-25T02:39:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.e.log 2023-08-25T02:39:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.4.log 2023-08-25T02:39:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.1.log 2023-08-25T02:39:45.431 INFO:teuthology.orchestra.run.smithi121.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-08-25T02:39:45.432 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-08-25T02:39:45.433 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-08-25T02:39:45.434 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.k.log 2023-08-25T02:39:45.434 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log 2023-08-25T02:39:45.435 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.3.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.g.log.gz 2023-08-25T02:39:45.435 INFO:teuthology.orchestra.run.smithi089.stderr: 94.6% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.a.log.gz 2023-08-25T02:39:45.435 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.x.log 2023-08-25T02:39:45.435 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.9.log 2023-08-25T02:39:45.436 INFO:teuthology.orchestra.run.smithi089.stderr: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.d.log.gz 2023-08-25T02:39:45.436 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log 2023-08-25T02:39:45.436 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.9.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-08-25T02:39:45.436 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.j.log 2023-08-25T02:39:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.a.log 2023-08-25T02:39:45.437 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.k.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.k.log.gz 2023-08-25T02:39:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.a.log: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.j.log.gz 2023-08-25T02:39:45.438 INFO:teuthology.orchestra.run.smithi121.stderr: 91.3% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log.gz 2023-08-25T02:39:45.439 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.7.log 2023-08-25T02:39:45.439 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.b.log 2023-08-25T02:39:45.440 INFO:teuthology.orchestra.run.smithi089.stderr: 91.4% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log.gz 2023-08-25T02:39:45.441 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.5.log 2023-08-25T02:39:45.441 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.2.log 2023-08-25T02:39:45.441 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.11.log 2023-08-25T02:39:45.441 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.z.log 2023-08-25T02:39:45.442 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.7.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.b.log 2023-08-25T02:39:45.443 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.8.log 2023-08-25T02:39:45.444 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.f.log 2023-08-25T02:39:45.444 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.c.log 2023-08-25T02:39:45.444 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.i.log 2023-08-25T02:39:45.445 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.c.log 2023-08-25T02:39:45.445 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.c.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log 2023-08-25T02:39:45.445 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.l.log 2023-08-25T02:39:45.446 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log: /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.l.log.gz 2023-08-25T02:39:45.446 INFO:teuthology.orchestra.run.smithi122.stderr: 91.7% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.z.log.gz 2023-08-25T02:39:45.452 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.y.log 2023-08-25T02:39:45.465 INFO:teuthology.orchestra.run.smithi122.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2023-08-25T02:39:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.y.log: 91.7% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.y.log.gz 2023-08-25T02:39:45.629 INFO:teuthology.orchestra.run.smithi089.stderr: 89.8% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mgr.x.log.gz 2023-08-25T02:39:45.694 INFO:teuthology.orchestra.run.smithi121.stderr: 94.3% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.e.log.gz 2023-08-25T02:39:45.754 INFO:teuthology.orchestra.run.smithi122.stderr: 94.0% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.f.log.gz 2023-08-25T02:39:45.881 INFO:teuthology.orchestra.run.smithi121.stderr: 91.9% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.b.log.gz 2023-08-25T02:39:46.156 INFO:teuthology.orchestra.run.smithi122.stderr: 95.3% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.c.log.gz 2023-08-25T02:39:46.270 INFO:teuthology.orchestra.run.smithi089.stderr: 90.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.a.log.gz 2023-08-25T02:39:46.370 INFO:teuthology.orchestra.run.smithi122.stderr: 92.0% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mon.c.log.gz 2023-08-25T02:39:46.611 INFO:teuthology.orchestra.run.smithi121.stderr: 94.2% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.1.log.gz 2023-08-25T02:39:46.612 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.6.log.gz 2023-08-25T02:39:46.671 INFO:teuthology.orchestra.run.smithi122.stderr: 91.2% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-volume.log.gz 2023-08-25T02:39:46.826 INFO:teuthology.orchestra.run.smithi089.stderr: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.3.log.gz 2023-08-25T02:39:46.875 INFO:teuthology.orchestra.run.smithi089.stderr: 94.4% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.9.log.gz 2023-08-25T02:39:46.933 INFO:teuthology.orchestra.run.smithi089.stderr: 94.9% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.0.log.gz 2023-08-25T02:39:46.935 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-08-25T02:39:46.936 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m1.523s 2023-08-25T02:39:46.936 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m6.339s 2023-08-25T02:39:46.936 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.284s 2023-08-25T02:39:47.095 INFO:teuthology.orchestra.run.smithi121.stderr: 94.2% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.7.log.gz 2023-08-25T02:39:47.335 INFO:teuthology.orchestra.run.smithi122.stderr: 93.7% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.8.log.gz 2023-08-25T02:39:47.383 INFO:teuthology.orchestra.run.smithi121.stderr: 94.4% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.4.log.gz 2023-08-25T02:39:47.596 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.10.log.gz 2023-08-25T02:39:47.611 INFO:teuthology.orchestra.run.smithi122.stderr: 94.6% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.2.log.gz 2023-08-25T02:39:47.627 INFO:teuthology.orchestra.run.smithi122.stderr: 94.0% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.5.log.gz 2023-08-25T02:39:47.734 INFO:teuthology.orchestra.run.smithi122.stderr: 94.5% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-osd.11.log.gz 2023-08-25T02:39:48.164 INFO:teuthology.orchestra.run.smithi121.stderr: 95.0% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.h.log.gz 2023-08-25T02:39:51.517 INFO:teuthology.orchestra.run.smithi121.stderr: 96.1% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.b.log.gz 2023-08-25T02:39:51.519 INFO:teuthology.orchestra.run.smithi121.stderr: 2023-08-25T02:39:51.519 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m6.104s 2023-08-25T02:39:51.519 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m15.916s 2023-08-25T02:39:51.520 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.568s 2023-08-25T02:42:55.791 INFO:teuthology.orchestra.run.smithi122.stderr: 97.7% -- replaced with /var/log/ceph/7eb69934-42ed-11ee-9b3d-001a4aab830c/ceph-mds.i.log.gz 2023-08-25T02:42:55.793 INFO:teuthology.orchestra.run.smithi122.stderr: 2023-08-25T02:42:55.794 INFO:teuthology.orchestra.run.smithi122.stderr:real 3m10.379s 2023-08-25T02:42:55.794 INFO:teuthology.orchestra.run.smithi122.stderr:user 2m57.210s 2023-08-25T02:42:55.794 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m8.210s 2023-08-25T02:42:55.795 INFO:tasks.cephadm:Archiving logs... 2023-08-25T02:42:55.795 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi089/log 2023-08-25T02:42:55.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-25T02:42:56.537 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi121/log 2023-08-25T02:42:56.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-25T02:42:58.084 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi122/log 2023-08-25T02:42:58.086 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-25T02:43:09.741 INFO:tasks.cephadm:Removing cluster... 2023-08-25T02:43:09.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force 2023-08-25T02:43:10.043 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:43:10.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force 2023-08-25T02:43:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:43:11.273 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eb69934-42ed-11ee-9b3d-001a4aab830c --force 2023-08-25T02:43:11.940 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 7eb69934-42ed-11ee-9b3d-001a4aab830c 2023-08-25T02:43:12.499 INFO:tasks.cephadm:Removing cephadm ... 2023-08-25T02:43:12.500 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2023-08-25T02:43:12.523 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2023-08-25T02:43:12.545 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2023-08-25T02:43:12.564 INFO:tasks.cephadm:Teardown complete 2023-08-25T02:43:12.566 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-08-25T02:43:12.577 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-08-25T02:43:12.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-25T02:43:12.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-25T02:43:12.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-25T02:43:12.762 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-08-25T02:43:12.773 INFO:teuthology.task.clock:Checking final clock skew... 2023-08-25T02:43:12.774 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:43:12.777 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:43:12.779 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T02:43:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2023-08-25T02:43:12.796 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2023-08-25T02:43:12.797 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-08-25T02:43:12.798 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:43:12.799 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2023-08-25T02:43:12.799 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +1604us[+1631us] +/- 76ms 2023-08-25T02:43:12.800 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -2018us[-1991us] +/- 57ms 2023-08-25T02:43:12.800 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +1316us[+1343us] +/- 84ms 2023-08-25T02:43:12.801 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:43:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:43:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-08-25T02:43:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +1672us[+1672us] +/- 77ms 2023-08-25T02:43:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -1985us[-2008us] +/- 57ms 2023-08-25T02:43:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1367us[+1344us] +/- 84ms 2023-08-25T02:43:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:43:12.823 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T02:43:12.824 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2023-08-25T02:43:12.824 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +1621us[+1621us] +/- 76ms 2023-08-25T02:43:12.825 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -2032us[-1915us] +/- 57ms 2023-08-25T02:43:12.825 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +1317us[+1434us] +/- 84ms 2023-08-25T02:43:12.825 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T02:43:12.826 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-08-25T02:43:12.842 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-08-25T02:43:12.843 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-08-25T02:43:12.923 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-25T02:43:13.029 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-25T02:43:13.132 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-25T02:43:13.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:43:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:13.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:43:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:13.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T02:43:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:13.379 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-08-25T02:43:13.390 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-08-25T02:43:13.401 INFO:teuthology.task.internal:Duration was 2428.597293 seconds 2023-08-25T02:43:13.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-08-25T02:43:13.419 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-08-25T02:43:13.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-25T02:43:13.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-25T02:43:13.426 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-25T02:43:13.469 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:43:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:43:13.514 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T02:43:13.762 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-08-25T02:43:13.763 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2023-08-25T02:43:13.763 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-25T02:43:13.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2023-08-25T02:43:13.786 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-25T02:43:13.809 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2023-08-25T02:43:13.809 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-25T02:43:13.839 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-08-25T02:43:13.840 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-25T02:43:13.842 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-25T02:43:13.851 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-25T02:43:13.951 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-08-25T02:43:13.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-25T02:43:13.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-25T02:43:13.966 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-25T02:43:14.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-08-25T02:43:14.088 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-25T02:43:14.088 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-25T02:43:14.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-25T02:43:14.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-25T02:43:14.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-08-25T02:43:14.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.241 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2023-08-25T02:43:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2023-08-25T02:43:14.261 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2023-08-25T02:43:14.297 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:14.316 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:14.356 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-25T02:43:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T02:43:14.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-08-25T02:43:14.391 INFO:teuthology.task.internal:Transferring archived files... 2023-08-25T02:43:14.394 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi089 2023-08-25T02:43:14.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-25T02:43:14.439 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi121 2023-08-25T02:43:14.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-25T02:43:14.485 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922/remote/smithi122 2023-08-25T02:43:14.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-25T02:43:14.525 INFO:teuthology.task.internal:Removing archive directory... 2023-08-25T02:43:14.526 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-25T02:43:14.528 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-25T02:43:14.531 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-25T02:43:14.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-08-25T02:43:14.590 INFO:teuthology.task.internal:Not uploading archives. 2023-08-25T02:43:14.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-08-25T02:43:14.602 INFO:teuthology.task.internal:Tidying up after the test... 2023-08-25T02:43:14.602 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-25T02:43:14.605 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-25T02:43:14.607 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-25T02:43:14.619 INFO:teuthology.orchestra.run.smithi089.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 25 02:43 /home/ubuntu/cephtest 2023-08-25T02:43:14.621 INFO:teuthology.orchestra.run.smithi121.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 25 02:43 /home/ubuntu/cephtest 2023-08-25T02:43:14.632 INFO:teuthology.orchestra.run.smithi122.stdout: 262181 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 25 02:43 /home/ubuntu/cephtest 2023-08-25T02:43:14.633 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-08-25T02:43:14.645 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-25T02:43:14.667 INFO:teuthology.nuke:Checking targets against current locks 2023-08-25T02:43:14.693 DEBUG:teuthology.nuke:shortname: smithi089 2023-08-25T02:43:14.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T02:43:14.713 DEBUG:teuthology.nuke:shortname: smithi121 2023-08-25T02:43:14.714 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T02:43:14.717 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.093492', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-25T02:43:14.734 DEBUG:teuthology.nuke:shortname: smithi122 2023-08-25T02:43:14.735 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T02:43:14.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.094237', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-08-25T02:43:14.754 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_01:50:32-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7378922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 01:53:38.092559', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-25T02:43:14.767 INFO:teuthology.orchestra.console.smithi089:Power off 2023-08-25T02:43:14.769 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-25T02:43:14.798 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2023-08-25T02:43:14.799 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:14.820 INFO:teuthology.orchestra.console.smithi121:Power off 2023-08-25T02:43:14.821 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-25T02:43:14.838 INFO:teuthology.orchestra.console.smithi122:Power off 2023-08-25T02:43:14.839 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-25T02:43:14.851 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-08-25T02:43:14.852 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:14.861 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-08-25T02:43:14.863 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2023-08-25T02:43:14.864 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:14.877 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-25T02:43:14.889 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2023-08-25T02:43:18.863 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:18.878 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:18.890 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:18.976 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-08-25T02:43:18.990 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-25T02:43:19.004 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2023-08-25T02:43:22.980 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:22.994 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:23.007 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T02:43:23.096 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2023-08-25T02:43:23.108 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-25T02:43:23.122 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2023-08-25T02:43:23.197 INFO:teuthology.orchestra.console.smithi089:Power off completed 2023-08-25T02:43:23.209 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-08-25T02:43:23.224 INFO:teuthology.orchestra.console.smithi122:Power off completed 2023-08-25T02:43:23.332 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} duration: 2428.597293138504 failure_reason: 'Command failed on smithi089 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''sudo -u postgres -- pgbench -s 500 -i''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=039c794437cd4bdda52749a26240ac89 status: fail success: false 2023-08-25T02:43:23.333 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-25T02:43:23.413 INFO:teuthology.run:FAIL